2024-04-26T02:07:35.803 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T02:07:35.803 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} --owner scheduled_yuriw@teuthology --name yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674282 -- /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674282/orig.config.yaml 2024-04-26T02:07:35.839 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T02:07:35.912 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674282 branch: wip-yuri2-testing-2024-04-24-0914-squid description: rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} email: yweinste@redhat.com first_in_suite: false job_id: '7674282' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri2-testing-2024-04-24-0914-squid ceph: conf: client: debug cephsqlite: 20 debug client: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 1500 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - POOL_APP_NOT_ENABLED - do not have an application enabled mon_bind_msgr2: false sha1: ddeaa91797339a19172c3036ff48cca58c12f448 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: ddeaa91797339a19172c3036ff48cca58c12f448 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri2-testing-2024-04-24-0914-squid sha1: ddeaa91797339a19172c3036ff48cca58c12f448 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 1848 sha1: ddeaa91797339a19172c3036ff48cca58c12f448 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri2-testing-2024-04-24-0914-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ddeaa91797339a19172c3036ff48cca58c12f448 targets: smithi087.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMyKkTW4yPxjq/fNiQrlOxp4P5aAVaV2x3BU5ji/rAibHmTisV0URzcbl8/WtBPYn+9IfByO+47YWaythDc/tMo= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPPjC4Gp4IcpOCh1ISncnKzkhVM1UAGkyhgg83Xam6I3XhAPDIWfBR5lFiXozObI3bKw86iNzsYG9BStDKdUdsc= tasks: - install: extra_system_packages: deb: - sqlite3 rpm: - sqlite-devel - ceph: null - exec: client.0: - ceph osd pool create cephsqlite - ceph auth get-or-create client.libcephsqlite mon 'profile simple-rados-client-with-blocklist' osd 'allow rwx pool=cephsqlite' >> /etc/ceph/ceph.keyring - exec: client.0: - ceph_test_libcephsqlite --id libcephsqlite --no-log-to-stderr - workunit: clients: client.0: - rados/test_libcephsqlite.sh cephsqlite env: CEPH_ARGS: --id libcephsqlite --no-log-to-stderr teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_01:23:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T02:07:35.913 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa; will attempt to use it 2024-04-26T02:07:35.913 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa/tasks 2024-04-26T02:07:35.913 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T02:07:35.915 INFO:teuthology.task.internal:Checking packages... 2024-04-26T02:07:35.936 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'ddeaa91797339a19172c3036ff48cca58c12f448' 2024-04-26T02:07:35.937 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T02:07:35.937 INFO:teuthology.packaging:ref: None 2024-04-26T02:07:35.937 INFO:teuthology.packaging:tag: None 2024-04-26T02:07:35.937 INFO:teuthology.packaging:branch: wip-yuri2-testing-2024-04-24-0914-squid 2024-04-26T02:07:35.937 INFO:teuthology.packaging:sha1: ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T02:07:35.937 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri2-testing-2024-04-24-0914-squid 2024-04-26T02:07:36.079 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2457-gddeaa917-1jammy 2024-04-26T02:07:36.081 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T02:07:36.091 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T02:07:36.091 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T02:07:36.146 INFO:teuthology.task.internal:Saving configuration 2024-04-26T02:07:36.158 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T02:07:36.204 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T02:07:36.229 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674282', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 02:01:55.168831', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEJsucyXXfUimYjS0HfqgNa4w2jwkboCh3QHLQYsl4eS'} 2024-04-26T02:07:36.250 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674282', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 02:01:55.167064', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA6tTshDmKPwKpU/1x4G8bFYXZhPw+miUBy9WRSeul3q'} 2024-04-26T02:07:36.250 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T02:07:36.258 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-04-26T02:07:36.258 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-26T02:07:36.258 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T02:07:36.382 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f978440c3a0>, signals=[15]) 2024-04-26T02:07:36.383 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T02:07:36.389 INFO:teuthology.task.internal:Opening connections... 2024-04-26T02:07:36.389 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2024-04-26T02:07:36.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T02:07:36.477 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-04-26T02:07:36.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T02:07:36.558 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T02:07:36.564 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2024-04-26T02:07:36.569 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2024-04-26T02:07:36.570 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2024-04-26T02:07:36.616 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T02:07:36.616 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="Ubuntu" 2024-04-26T02:07:36.616 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="22.04" 2024-04-26T02:07:36.616 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T02:07:36.616 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_CODENAME=jammy 2024-04-26T02:07:36.616 INFO:teuthology.orchestra.run.smithi087.stdout:ID=ubuntu 2024-04-26T02:07:36.616 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE=debian 2024-04-26T02:07:36.616 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T02:07:36.616 INFO:teuthology.orchestra.run.smithi087.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T02:07:36.616 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T02:07:36.616 INFO:teuthology.orchestra.run.smithi087.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T02:07:36.616 INFO:teuthology.orchestra.run.smithi087.stdout:UBUNTU_CODENAME=jammy 2024-04-26T02:07:36.617 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2024-04-26T02:07:36.640 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-04-26T02:07:36.645 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-04-26T02:07:36.645 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-04-26T02:07:36.693 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T02:07:36.693 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="Ubuntu" 2024-04-26T02:07:36.693 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="22.04" 2024-04-26T02:07:36.693 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T02:07:36.693 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_CODENAME=jammy 2024-04-26T02:07:36.693 INFO:teuthology.orchestra.run.smithi122.stdout:ID=ubuntu 2024-04-26T02:07:36.693 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE=debian 2024-04-26T02:07:36.693 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T02:07:36.693 INFO:teuthology.orchestra.run.smithi122.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T02:07:36.693 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T02:07:36.694 INFO:teuthology.orchestra.run.smithi122.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T02:07:36.694 INFO:teuthology.orchestra.run.smithi122.stdout:UBUNTU_CODENAME=jammy 2024-04-26T02:07:36.694 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-04-26T02:07:36.715 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T02:07:36.724 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T02:07:36.781 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T02:07:36.782 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2024-04-26T02:07:36.784 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-04-26T02:07:36.789 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T02:07:36.868 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T02:07:36.868 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2024-04-26T02:07:36.871 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-04-26T02:07:36.882 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T02:07:36.928 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T02:07:36.947 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-26T02:07:36.947 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-26T02:07:36.948 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T02:07:36.948 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T02:07:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:07:36.953 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-04-26T02:07:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:07:36.954 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-04-26T02:07:37.001 INFO:teuthology.orchestra.run.smithi087.stdout:5.15.0-56-generic 2024-04-26T02:07:37.001 INFO:teuthology.task.kernel:Running kernel on smithi087: 5.15.0-56-generic 2024-04-26T02:07:37.001 DEBUG:teuthology.orchestra.run.smithi087:> sudo apt-get clean 2024-04-26T02:07:37.002 INFO:teuthology.orchestra.run.smithi122.stdout:5.15.0-56-generic 2024-04-26T02:07:37.002 INFO:teuthology.task.kernel:Running kernel on smithi122: 5.15.0-56-generic 2024-04-26T02:07:37.002 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get clean 2024-04-26T02:07:37.131 DEBUG:teuthology.orchestra.run.smithi087:> sudo apt-get update 2024-04-26T02:07:37.139 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get update 2024-04-26T02:07:37.304 INFO:teuthology.orchestra.run.smithi122.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T02:07:37.406 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T02:07:37.411 INFO:teuthology.orchestra.run.smithi087.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T02:07:37.440 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T02:07:37.499 INFO:teuthology.orchestra.run.smithi122.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T02:07:37.509 INFO:teuthology.orchestra.run.smithi087.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T02:07:37.547 INFO:teuthology.orchestra.run.smithi122.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T02:07:37.737 INFO:teuthology.orchestra.run.smithi122.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T02:07:37.834 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T02:07:37.840 INFO:teuthology.orchestra.run.smithi122.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T02:07:37.840 INFO:teuthology.orchestra.run.smithi122.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T02:07:37.845 INFO:teuthology.orchestra.run.smithi087.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T02:07:37.862 INFO:teuthology.orchestra.run.smithi122.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T02:07:37.881 INFO:teuthology.orchestra.run.smithi122.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T02:07:37.881 INFO:teuthology.orchestra.run.smithi122.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T02:07:37.884 INFO:teuthology.orchestra.run.smithi122.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T02:07:37.884 INFO:teuthology.orchestra.run.smithi122.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T02:07:37.888 INFO:teuthology.orchestra.run.smithi087.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T02:07:37.900 INFO:teuthology.orchestra.run.smithi122.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T02:07:37.910 INFO:teuthology.orchestra.run.smithi122.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T02:07:37.912 INFO:teuthology.orchestra.run.smithi122.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T02:07:37.939 INFO:teuthology.orchestra.run.smithi122.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T02:07:37.939 INFO:teuthology.orchestra.run.smithi122.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T02:07:37.941 INFO:teuthology.orchestra.run.smithi122.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T02:07:37.941 INFO:teuthology.orchestra.run.smithi122.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T02:07:38.022 INFO:teuthology.orchestra.run.smithi122.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T02:07:38.115 INFO:teuthology.orchestra.run.smithi087.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T02:07:38.337 INFO:teuthology.orchestra.run.smithi122.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T02:07:38.377 INFO:teuthology.orchestra.run.smithi122.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T02:07:38.392 INFO:teuthology.orchestra.run.smithi122.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T02:07:38.393 INFO:teuthology.orchestra.run.smithi122.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T02:07:38.456 INFO:teuthology.orchestra.run.smithi122.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T02:07:38.457 INFO:teuthology.orchestra.run.smithi122.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T02:07:38.464 INFO:teuthology.orchestra.run.smithi122.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T02:07:38.629 INFO:teuthology.orchestra.run.smithi122.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T02:07:39.098 INFO:teuthology.orchestra.run.smithi087.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T02:07:39.219 INFO:teuthology.orchestra.run.smithi122.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T02:07:39.331 INFO:teuthology.orchestra.run.smithi122.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T02:07:39.353 INFO:teuthology.orchestra.run.smithi122.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T02:07:39.353 INFO:teuthology.orchestra.run.smithi122.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T02:07:39.358 INFO:teuthology.orchestra.run.smithi122.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T02:07:39.358 INFO:teuthology.orchestra.run.smithi122.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T02:07:39.359 INFO:teuthology.orchestra.run.smithi122.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T02:07:39.359 INFO:teuthology.orchestra.run.smithi122.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T02:07:39.363 INFO:teuthology.orchestra.run.smithi122.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T02:07:39.368 INFO:teuthology.orchestra.run.smithi122.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T02:07:39.535 INFO:teuthology.orchestra.run.smithi122.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T02:07:39.617 INFO:teuthology.orchestra.run.smithi122.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T02:07:39.699 INFO:teuthology.orchestra.run.smithi122.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T02:07:39.780 INFO:teuthology.orchestra.run.smithi122.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T02:07:39.810 INFO:teuthology.orchestra.run.smithi122.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T02:07:39.816 INFO:teuthology.orchestra.run.smithi087.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T02:07:40.159 INFO:teuthology.orchestra.run.smithi087.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T02:07:40.170 INFO:teuthology.orchestra.run.smithi087.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T02:07:41.344 INFO:teuthology.orchestra.run.smithi087.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T02:07:42.412 INFO:teuthology.orchestra.run.smithi087.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T02:07:42.526 INFO:teuthology.orchestra.run.smithi087.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T02:07:42.707 INFO:teuthology.orchestra.run.smithi087.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T02:07:43.059 INFO:teuthology.orchestra.run.smithi087.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T02:07:43.059 INFO:teuthology.orchestra.run.smithi087.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T02:07:43.221 INFO:teuthology.orchestra.run.smithi087.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T02:07:43.266 INFO:teuthology.orchestra.run.smithi087.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T02:07:43.719 INFO:teuthology.orchestra.run.smithi087.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T02:07:44.505 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 13.2 MB in 3s (5,118 kB/s) 2024-04-26T02:07:44.949 INFO:teuthology.orchestra.run.smithi087.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T02:07:45.352 INFO:teuthology.orchestra.run.smithi087.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T02:07:45.358 INFO:teuthology.orchestra.run.smithi087.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T02:07:45.359 INFO:teuthology.orchestra.run.smithi087.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T02:07:45.374 INFO:teuthology.orchestra.run.smithi087.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T02:07:45.376 INFO:teuthology.orchestra.run.smithi087.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T02:07:45.831 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2024-04-26T02:07:45.847 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T02:07:45.906 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2024-04-26T02:07:46.105 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2024-04-26T02:07:46.106 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2024-04-26T02:07:46.271 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T02:07:46.271 INFO:teuthology.orchestra.run.smithi122.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T02:07:46.271 INFO:teuthology.orchestra.run.smithi122.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T02:07:46.271 INFO:teuthology.orchestra.run.smithi122.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T02:07:46.271 INFO:teuthology.orchestra.run.smithi122.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T02:07:46.271 INFO:teuthology.orchestra.run.smithi122.stdout: python2.7-minimal 2024-04-26T02:07:46.271 INFO:teuthology.orchestra.run.smithi122.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T02:07:46.271 INFO:teuthology.orchestra.run.smithi122.stdout:The following additional packages will be installed: 2024-04-26T02:07:46.271 INFO:teuthology.orchestra.run.smithi122.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T02:07:46.271 INFO:teuthology.orchestra.run.smithi122.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T02:07:46.272 INFO:teuthology.orchestra.run.smithi122.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T02:07:46.272 INFO:teuthology.orchestra.run.smithi122.stdout:Suggested packages: 2024-04-26T02:07:46.272 INFO:teuthology.orchestra.run.smithi122.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T02:07:46.272 INFO:teuthology.orchestra.run.smithi122.stdout:Recommended packages: 2024-04-26T02:07:46.272 INFO:teuthology.orchestra.run.smithi122.stdout: thermald 2024-04-26T02:07:46.321 INFO:teuthology.orchestra.run.smithi122.stdout:The following NEW packages will be installed: 2024-04-26T02:07:46.321 INFO:teuthology.orchestra.run.smithi122.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T02:07:46.321 INFO:teuthology.orchestra.run.smithi122.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T02:07:46.322 INFO:teuthology.orchestra.run.smithi122.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T02:07:46.323 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages will be upgraded: 2024-04-26T02:07:46.323 INFO:teuthology.orchestra.run.smithi122.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T02:07:46.379 INFO:teuthology.orchestra.run.smithi122.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T02:07:46.380 INFO:teuthology.orchestra.run.smithi122.stdout:Need to get 113 MB of archives. 2024-04-26T02:07:46.380 INFO:teuthology.orchestra.run.smithi122.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T02:07:46.380 INFO:teuthology.orchestra.run.smithi122.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T02:07:46.960 INFO:teuthology.orchestra.run.smithi122.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T02:07:47.150 INFO:teuthology.orchestra.run.smithi087.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T02:07:47.172 INFO:teuthology.orchestra.run.smithi122.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T02:07:47.228 INFO:teuthology.orchestra.run.smithi087.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T02:07:48.017 INFO:teuthology.orchestra.run.smithi087.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T02:07:48.183 INFO:teuthology.orchestra.run.smithi087.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T02:07:48.412 INFO:teuthology.orchestra.run.smithi122.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T02:07:48.412 INFO:teuthology.orchestra.run.smithi122.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T02:07:48.412 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T02:07:48.807 INFO:teuthology.orchestra.run.smithi087.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T02:07:48.822 INFO:teuthology.orchestra.run.smithi122.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T02:07:48.925 INFO:teuthology.orchestra.run.smithi122.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T02:07:49.161 INFO:teuthology.orchestra.run.smithi087.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T02:07:49.206 INFO:teuthology.orchestra.run.smithi087.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T02:07:49.208 INFO:teuthology.orchestra.run.smithi087.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T02:07:49.212 INFO:teuthology.orchestra.run.smithi087.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T02:07:49.212 INFO:teuthology.orchestra.run.smithi087.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T02:07:49.213 INFO:teuthology.orchestra.run.smithi087.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T02:07:49.214 INFO:teuthology.orchestra.run.smithi087.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T02:07:49.221 INFO:teuthology.orchestra.run.smithi087.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T02:07:49.226 INFO:teuthology.orchestra.run.smithi087.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T02:07:49.315 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 113 MB in 3s (43.7 MB/s) 2024-04-26T02:07:49.391 INFO:teuthology.orchestra.run.smithi087.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T02:07:49.474 INFO:teuthology.orchestra.run.smithi087.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T02:07:49.491 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T02:07:49.581 INFO:teuthology.orchestra.run.smithi087.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T02:07:49.614 INFO:teuthology.orchestra.run.smithi087.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T02:07:49.675 INFO:teuthology.orchestra.run.smithi087.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T02:07:51.485 INFO:teuthology.orchestra.run.smithi122.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T02:07:51.489 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T02:07:51.689 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:07:53.866 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T02:07:53.870 INFO:teuthology.orchestra.run.smithi087.stdout:Fetched 13.2 MB in 12s (1,061 kB/s) 2024-04-26T02:07:53.887 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T02:07:53.984 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:07:54.452 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T02:07:54.472 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T02:07:54.510 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:07:55.252 INFO:teuthology.orchestra.run.smithi087.stdout:Reading package lists... 2024-04-26T02:07:55.270 DEBUG:teuthology.orchestra.run.smithi087:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T02:07:55.334 INFO:teuthology.orchestra.run.smithi087.stdout:Reading package lists... 2024-04-26T02:07:55.541 INFO:teuthology.orchestra.run.smithi087.stdout:Building dependency tree... 2024-04-26T02:07:55.541 INFO:teuthology.orchestra.run.smithi087.stdout:Reading state information... 2024-04-26T02:07:55.720 INFO:teuthology.orchestra.run.smithi087.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T02:07:55.720 INFO:teuthology.orchestra.run.smithi087.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T02:07:55.720 INFO:teuthology.orchestra.run.smithi087.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T02:07:55.721 INFO:teuthology.orchestra.run.smithi087.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T02:07:55.721 INFO:teuthology.orchestra.run.smithi087.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T02:07:55.721 INFO:teuthology.orchestra.run.smithi087.stdout: python2.7-minimal 2024-04-26T02:07:55.721 INFO:teuthology.orchestra.run.smithi087.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T02:07:55.722 INFO:teuthology.orchestra.run.smithi087.stdout:The following additional packages will be installed: 2024-04-26T02:07:55.722 INFO:teuthology.orchestra.run.smithi087.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T02:07:55.722 INFO:teuthology.orchestra.run.smithi087.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T02:07:55.723 INFO:teuthology.orchestra.run.smithi087.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T02:07:55.724 INFO:teuthology.orchestra.run.smithi087.stdout:Suggested packages: 2024-04-26T02:07:55.724 INFO:teuthology.orchestra.run.smithi087.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T02:07:55.724 INFO:teuthology.orchestra.run.smithi087.stdout:Recommended packages: 2024-04-26T02:07:55.724 INFO:teuthology.orchestra.run.smithi087.stdout: thermald 2024-04-26T02:07:55.776 INFO:teuthology.orchestra.run.smithi087.stdout:The following NEW packages will be installed: 2024-04-26T02:07:55.776 INFO:teuthology.orchestra.run.smithi087.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T02:07:55.776 INFO:teuthology.orchestra.run.smithi087.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T02:07:55.777 INFO:teuthology.orchestra.run.smithi087.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T02:07:55.778 INFO:teuthology.orchestra.run.smithi087.stdout:The following packages will be upgraded: 2024-04-26T02:07:55.778 INFO:teuthology.orchestra.run.smithi087.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T02:07:55.962 INFO:teuthology.orchestra.run.smithi087.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T02:07:55.962 INFO:teuthology.orchestra.run.smithi087.stdout:Need to get 113 MB of archives. 2024-04-26T02:07:55.962 INFO:teuthology.orchestra.run.smithi087.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T02:07:55.962 INFO:teuthology.orchestra.run.smithi087.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T02:08:01.355 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T02:08:01.455 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T02:08:01.763 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T02:08:01.892 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T02:08:02.229 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T02:08:02.251 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T02:08:02.295 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T02:08:07.476 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T02:08:07.511 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T02:08:07.593 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:08:10.304 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T02:08:10.429 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T02:08:10.754 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T02:08:10.875 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:08:11.002 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T02:08:11.130 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:08:12.448 INFO:teuthology.orchestra.run.smithi122.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T02:08:12.448 INFO:teuthology.orchestra.run.smithi122.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T02:08:12.448 INFO:teuthology.orchestra.run.smithi122.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T02:08:12.448 INFO:teuthology.orchestra.run.smithi122.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T02:08:12.616 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:08:14.426 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:08:15.819 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T02:08:15.945 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T02:08:16.054 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:08:16.121 INFO:teuthology.orchestra.run.smithi122.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T02:08:16.121 INFO:teuthology.orchestra.run.smithi122.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T02:08:30.093 INFO:teuthology.orchestra.run.smithi122.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T02:08:30.094 INFO:teuthology.orchestra.run.smithi122.stdout:Sourcing file `/etc/default/grub' 2024-04-26T02:08:30.104 INFO:teuthology.orchestra.run.smithi122.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T02:08:30.131 INFO:teuthology.orchestra.run.smithi122.stdout:Generating grub configuration file ... 2024-04-26T02:08:30.395 INFO:teuthology.orchestra.run.smithi122.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T02:08:30.408 INFO:teuthology.orchestra.run.smithi122.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T02:08:30.705 INFO:teuthology.orchestra.run.smithi122.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T02:08:30.711 INFO:teuthology.orchestra.run.smithi122.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T02:08:30.777 INFO:teuthology.orchestra.run.smithi122.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T02:08:30.782 INFO:teuthology.orchestra.run.smithi122.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T02:08:30.931 INFO:teuthology.orchestra.run.smithi122.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T02:08:30.931 INFO:teuthology.orchestra.run.smithi122.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T02:08:30.931 INFO:teuthology.orchestra.run.smithi122.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T02:08:30.975 INFO:teuthology.orchestra.run.smithi122.stdout:done 2024-04-26T02:08:31.443 DEBUG:teuthology.orchestra.run.smithi122:> dpkg -s linux-image-generic 2024-04-26T02:08:31.463 INFO:teuthology.orchestra.run.smithi122.stdout:Package: linux-image-generic 2024-04-26T02:08:31.464 INFO:teuthology.orchestra.run.smithi122.stdout:Status: install ok installed 2024-04-26T02:08:31.464 INFO:teuthology.orchestra.run.smithi122.stdout:Priority: optional 2024-04-26T02:08:31.464 INFO:teuthology.orchestra.run.smithi122.stdout:Section: kernel 2024-04-26T02:08:31.464 INFO:teuthology.orchestra.run.smithi122.stdout:Installed-Size: 21 2024-04-26T02:08:31.464 INFO:teuthology.orchestra.run.smithi122.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T02:08:31.464 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: amd64 2024-04-26T02:08:31.464 INFO:teuthology.orchestra.run.smithi122.stdout:Source: linux-meta 2024-04-26T02:08:31.464 INFO:teuthology.orchestra.run.smithi122.stdout:Version: 5.15.0.105.102 2024-04-26T02:08:31.464 INFO:teuthology.orchestra.run.smithi122.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T02:08:31.464 INFO:teuthology.orchestra.run.smithi122.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T02:08:31.464 INFO:teuthology.orchestra.run.smithi122.stdout:Recommends: thermald 2024-04-26T02:08:31.464 INFO:teuthology.orchestra.run.smithi122.stdout:Description: Generic Linux kernel image 2024-04-26T02:08:31.464 INFO:teuthology.orchestra.run.smithi122.stdout: This package will always depend on the latest generic kernel image 2024-04-26T02:08:31.465 INFO:teuthology.orchestra.run.smithi122.stdout: available. 2024-04-26T02:08:31.465 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T02:08:31.465 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T02:08:31.465 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T02:08:31.465 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-26T02:08:31.465 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T02:08:31.465 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get clean 2024-04-26T02:08:31.630 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get update 2024-04-26T02:08:31.783 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T02:08:31.783 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T02:08:31.795 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T02:08:31.829 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T02:08:33.185 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2024-04-26T02:08:33.203 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T02:08:33.262 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2024-04-26T02:08:33.463 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2024-04-26T02:08:33.463 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2024-04-26T02:08:33.639 INFO:teuthology.orchestra.run.smithi122.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T02:08:33.639 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T02:08:33.639 INFO:teuthology.orchestra.run.smithi122.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T02:08:33.639 INFO:teuthology.orchestra.run.smithi122.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T02:08:33.640 INFO:teuthology.orchestra.run.smithi122.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T02:08:33.640 INFO:teuthology.orchestra.run.smithi122.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T02:08:33.640 INFO:teuthology.orchestra.run.smithi122.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T02:08:33.640 INFO:teuthology.orchestra.run.smithi122.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T02:08:33.682 INFO:teuthology.orchestra.run.smithi122.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T02:08:33.684 DEBUG:teuthology.orchestra.run.smithi122:> dpkg -s linux-image-generic 2024-04-26T02:08:33.699 INFO:teuthology.orchestra.run.smithi122.stdout:Package: linux-image-generic 2024-04-26T02:08:33.699 INFO:teuthology.orchestra.run.smithi122.stdout:Status: install ok installed 2024-04-26T02:08:33.699 INFO:teuthology.orchestra.run.smithi122.stdout:Priority: optional 2024-04-26T02:08:33.699 INFO:teuthology.orchestra.run.smithi122.stdout:Section: kernel 2024-04-26T02:08:33.699 INFO:teuthology.orchestra.run.smithi122.stdout:Installed-Size: 21 2024-04-26T02:08:33.699 INFO:teuthology.orchestra.run.smithi122.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T02:08:33.699 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: amd64 2024-04-26T02:08:33.700 INFO:teuthology.orchestra.run.smithi122.stdout:Source: linux-meta 2024-04-26T02:08:33.700 INFO:teuthology.orchestra.run.smithi122.stdout:Version: 5.15.0.105.102 2024-04-26T02:08:33.700 INFO:teuthology.orchestra.run.smithi122.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T02:08:33.700 INFO:teuthology.orchestra.run.smithi122.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T02:08:33.700 INFO:teuthology.orchestra.run.smithi122.stdout:Recommends: thermald 2024-04-26T02:08:33.700 INFO:teuthology.orchestra.run.smithi122.stdout:Description: Generic Linux kernel image 2024-04-26T02:08:33.700 INFO:teuthology.orchestra.run.smithi122.stdout: This package will always depend on the latest generic kernel image 2024-04-26T02:08:33.700 INFO:teuthology.orchestra.run.smithi122.stdout: available. 2024-04-26T02:08:33.700 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-04-26T02:08:33.744 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.xu2ouI3Ttw 2024-04-26T02:08:33.745 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub/grub.cfg /tmp/tmp.xu2ouI3Ttw 2024-04-26T02:08:33.800 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.xu2ouI3Ttw 2024-04-26T02:08:33.926 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.xu2ouI3Ttw is 10KB 2024-04-26T02:08:33.943 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.xu2ouI3Ttw 2024-04-26T02:08:33.949 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T02:08:34.001 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:08:34.001 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T02:08:34.001 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T02:08:34.080 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T02:08:34.080 DEBUG:teuthology.orchestra.run.smithi122:> sudo update-grub 2024-04-26T02:08:35.324 INFO:teuthology.orchestra.run.smithi122.stderr:Sourcing file `/etc/default/grub' 2024-04-26T02:08:35.326 INFO:teuthology.orchestra.run.smithi122.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T02:08:35.329 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2024-04-26T02:08:35.560 INFO:teuthology.orchestra.run.smithi122.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T02:08:35.566 INFO:teuthology.orchestra.run.smithi122.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T02:08:35.795 INFO:teuthology.orchestra.run.smithi122.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T02:08:35.800 INFO:teuthology.orchestra.run.smithi122.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T02:08:35.840 INFO:teuthology.orchestra.run.smithi122.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T02:08:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T02:08:35.987 INFO:teuthology.orchestra.run.smithi122.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T02:08:35.987 INFO:teuthology.orchestra.run.smithi122.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T02:08:35.987 INFO:teuthology.orchestra.run.smithi122.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T02:08:36.019 INFO:teuthology.orchestra.run.smithi122.stderr:done 2024-04-26T02:08:36.022 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2024-04-26T02:08:41.762 INFO:teuthology.orchestra.run.smithi087.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T02:08:42.095 INFO:teuthology.orchestra.run.smithi087.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T02:08:43.045 INFO:teuthology.orchestra.run.smithi087.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T02:08:43.065 INFO:teuthology.orchestra.run.smithi087.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T02:08:43.085 INFO:teuthology.orchestra.run.smithi087.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T02:08:43.254 INFO:teuthology.orchestra.run.smithi087.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T02:08:43.308 INFO:teuthology.orchestra.run.smithi087.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T02:08:43.704 INFO:teuthology.orchestra.run.smithi087.stdout:Fetched 113 MB in 48s (2,384 kB/s) 2024-04-26T02:08:43.896 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T02:08:45.931 INFO:teuthology.orchestra.run.smithi087.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T02:08:45.935 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T02:08:46.025 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:08:48.217 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T02:08:48.238 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T02:08:48.328 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:08:48.837 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T02:08:48.860 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T02:08:48.887 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:08:55.965 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T02:08:56.110 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T02:08:56.442 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T02:08:56.538 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T02:08:56.866 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T02:08:56.877 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T02:08:56.916 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T02:09:02.697 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T02:09:02.733 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T02:09:02.763 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:09:05.508 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T02:09:05.633 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T02:09:06.007 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T02:09:06.024 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-26T02:09:06.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-26T02:09:06.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T02:09:06.129 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:09:06.239 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T02:09:06.350 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:09:07.660 INFO:teuthology.orchestra.run.smithi087.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T02:09:07.660 INFO:teuthology.orchestra.run.smithi087.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T02:09:07.661 INFO:teuthology.orchestra.run.smithi087.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T02:09:07.661 INFO:teuthology.orchestra.run.smithi087.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T02:09:07.827 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:09:09.254 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:09:10.622 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T02:09:10.748 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T02:09:10.874 INFO:teuthology.orchestra.run.smithi087.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T02:09:10.941 INFO:teuthology.orchestra.run.smithi087.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T02:09:10.941 INFO:teuthology.orchestra.run.smithi087.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T02:09:24.510 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-04-26T02:09:25.337 INFO:teuthology.orchestra.run.smithi087.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T02:09:25.337 INFO:teuthology.orchestra.run.smithi087.stdout:Sourcing file `/etc/default/grub' 2024-04-26T02:09:25.360 INFO:teuthology.orchestra.run.smithi087.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T02:09:25.385 INFO:teuthology.orchestra.run.smithi087.stdout:Generating grub configuration file ... 2024-04-26T02:09:25.656 INFO:teuthology.orchestra.run.smithi087.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T02:09:25.675 INFO:teuthology.orchestra.run.smithi087.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T02:09:26.021 INFO:teuthology.orchestra.run.smithi087.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T02:09:26.027 INFO:teuthology.orchestra.run.smithi087.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T02:09:26.092 INFO:teuthology.orchestra.run.smithi087.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T02:09:26.097 INFO:teuthology.orchestra.run.smithi087.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T02:09:26.245 INFO:teuthology.orchestra.run.smithi087.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T02:09:26.245 INFO:teuthology.orchestra.run.smithi087.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T02:09:26.245 INFO:teuthology.orchestra.run.smithi087.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T02:09:26.300 INFO:teuthology.orchestra.run.smithi087.stdout:done 2024-04-26T02:09:26.800 DEBUG:teuthology.orchestra.run.smithi087:> dpkg -s linux-image-generic 2024-04-26T02:09:26.823 INFO:teuthology.orchestra.run.smithi087.stdout:Package: linux-image-generic 2024-04-26T02:09:26.823 INFO:teuthology.orchestra.run.smithi087.stdout:Status: install ok installed 2024-04-26T02:09:26.823 INFO:teuthology.orchestra.run.smithi087.stdout:Priority: optional 2024-04-26T02:09:26.823 INFO:teuthology.orchestra.run.smithi087.stdout:Section: kernel 2024-04-26T02:09:26.823 INFO:teuthology.orchestra.run.smithi087.stdout:Installed-Size: 21 2024-04-26T02:09:26.824 INFO:teuthology.orchestra.run.smithi087.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T02:09:26.824 INFO:teuthology.orchestra.run.smithi087.stdout:Architecture: amd64 2024-04-26T02:09:26.824 INFO:teuthology.orchestra.run.smithi087.stdout:Source: linux-meta 2024-04-26T02:09:26.824 INFO:teuthology.orchestra.run.smithi087.stdout:Version: 5.15.0.105.102 2024-04-26T02:09:26.824 INFO:teuthology.orchestra.run.smithi087.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T02:09:26.824 INFO:teuthology.orchestra.run.smithi087.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T02:09:26.824 INFO:teuthology.orchestra.run.smithi087.stdout:Recommends: thermald 2024-04-26T02:09:26.824 INFO:teuthology.orchestra.run.smithi087.stdout:Description: Generic Linux kernel image 2024-04-26T02:09:26.824 INFO:teuthology.orchestra.run.smithi087.stdout: This package will always depend on the latest generic kernel image 2024-04-26T02:09:26.824 INFO:teuthology.orchestra.run.smithi087.stdout: available. 2024-04-26T02:09:26.824 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T02:09:26.824 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T02:09:26.824 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T02:09:26.825 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-26T02:09:26.825 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi087.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T02:09:26.825 DEBUG:teuthology.orchestra.run.smithi087:> sudo apt-get clean 2024-04-26T02:09:26.963 DEBUG:teuthology.orchestra.run.smithi087:> sudo apt-get update 2024-04-26T02:09:27.088 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T02:09:27.108 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T02:09:27.136 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T02:09:27.193 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T02:09:28.445 INFO:teuthology.orchestra.run.smithi087.stdout:Reading package lists... 2024-04-26T02:09:28.462 DEBUG:teuthology.orchestra.run.smithi087:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T02:09:28.525 INFO:teuthology.orchestra.run.smithi087.stdout:Reading package lists... 2024-04-26T02:09:28.735 INFO:teuthology.orchestra.run.smithi087.stdout:Building dependency tree... 2024-04-26T02:09:28.735 INFO:teuthology.orchestra.run.smithi087.stdout:Reading state information... 2024-04-26T02:09:28.916 INFO:teuthology.orchestra.run.smithi087.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T02:09:28.916 INFO:teuthology.orchestra.run.smithi087.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T02:09:28.916 INFO:teuthology.orchestra.run.smithi087.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T02:09:28.916 INFO:teuthology.orchestra.run.smithi087.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T02:09:28.916 INFO:teuthology.orchestra.run.smithi087.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T02:09:28.917 INFO:teuthology.orchestra.run.smithi087.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T02:09:28.917 INFO:teuthology.orchestra.run.smithi087.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T02:09:28.917 INFO:teuthology.orchestra.run.smithi087.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T02:09:28.960 INFO:teuthology.orchestra.run.smithi087.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T02:09:28.962 DEBUG:teuthology.orchestra.run.smithi087:> dpkg -s linux-image-generic 2024-04-26T02:09:28.974 INFO:teuthology.orchestra.run.smithi087.stdout:Package: linux-image-generic 2024-04-26T02:09:28.974 INFO:teuthology.orchestra.run.smithi087.stdout:Status: install ok installed 2024-04-26T02:09:28.975 INFO:teuthology.orchestra.run.smithi087.stdout:Priority: optional 2024-04-26T02:09:28.975 INFO:teuthology.orchestra.run.smithi087.stdout:Section: kernel 2024-04-26T02:09:28.975 INFO:teuthology.orchestra.run.smithi087.stdout:Installed-Size: 21 2024-04-26T02:09:28.975 INFO:teuthology.orchestra.run.smithi087.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T02:09:28.975 INFO:teuthology.orchestra.run.smithi087.stdout:Architecture: amd64 2024-04-26T02:09:28.975 INFO:teuthology.orchestra.run.smithi087.stdout:Source: linux-meta 2024-04-26T02:09:28.975 INFO:teuthology.orchestra.run.smithi087.stdout:Version: 5.15.0.105.102 2024-04-26T02:09:28.975 INFO:teuthology.orchestra.run.smithi087.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T02:09:28.975 INFO:teuthology.orchestra.run.smithi087.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T02:09:28.975 INFO:teuthology.orchestra.run.smithi087.stdout:Recommends: thermald 2024-04-26T02:09:28.975 INFO:teuthology.orchestra.run.smithi087.stdout:Description: Generic Linux kernel image 2024-04-26T02:09:28.975 INFO:teuthology.orchestra.run.smithi087.stdout: This package will always depend on the latest generic kernel image 2024-04-26T02:09:28.975 INFO:teuthology.orchestra.run.smithi087.stdout: available. 2024-04-26T02:09:28.976 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-04-26T02:09:29.022 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.ReL1OlIugA 2024-04-26T02:09:29.023 DEBUG:teuthology.orchestra.run.smithi087:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ReL1OlIugA 2024-04-26T02:09:29.077 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.ReL1OlIugA 2024-04-26T02:09:29.199 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.ReL1OlIugA is 10KB 2024-04-26T02:09:29.211 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.ReL1OlIugA 2024-04-26T02:09:29.216 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T02:09:29.269 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:09:29.270 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T02:09:29.270 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T02:09:29.348 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T02:09:29.348 DEBUG:teuthology.orchestra.run.smithi087:> sudo update-grub 2024-04-26T02:09:30.543 INFO:teuthology.orchestra.run.smithi087.stderr:Sourcing file `/etc/default/grub' 2024-04-26T02:09:30.544 INFO:teuthology.orchestra.run.smithi087.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T02:09:30.554 INFO:teuthology.orchestra.run.smithi087.stderr:Generating grub configuration file ... 2024-04-26T02:09:30.747 INFO:teuthology.orchestra.run.smithi087.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T02:09:30.753 INFO:teuthology.orchestra.run.smithi087.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T02:09:30.981 INFO:teuthology.orchestra.run.smithi087.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T02:09:30.986 INFO:teuthology.orchestra.run.smithi087.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T02:09:31.025 INFO:teuthology.orchestra.run.smithi087.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T02:09:31.030 INFO:teuthology.orchestra.run.smithi087.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T02:09:31.165 INFO:teuthology.orchestra.run.smithi087.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T02:09:31.165 INFO:teuthology.orchestra.run.smithi087.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T02:09:31.165 INFO:teuthology.orchestra.run.smithi087.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T02:09:31.194 INFO:teuthology.orchestra.run.smithi087.stderr:done 2024-04-26T02:09:31.203 DEBUG:teuthology.orchestra.run.smithi087:> sudo shutdown -r now 2024-04-26T02:09:33.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-26T02:09:33.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T02:09:36.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-04-26T02:09:48.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-26T02:09:48.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T02:10:01.218 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-26T02:10:01.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-26T02:10:01.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T02:10:19.550 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-04-26T02:10:20.436 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-04-26T02:10:21.184 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-26T02:10:21.184 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-26T02:10:21.185 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-04-26T02:10:21.237 INFO:teuthology.orchestra.run.smithi122.stdout:5.15.0-105-generic 2024-04-26T02:10:21.237 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T02:10:21.237 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T02:10:21.237 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T02:10:22.238 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-26T02:10:22.238 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T02:10:22.390 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-04-26T02:10:22.418 DEBUG:teuthology.parallel:result is None 2024-04-26T02:10:28.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-26T02:10:28.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T02:10:31.618 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-04-26T02:10:43.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-26T02:10:43.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T02:11:02.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-04-26T02:11:17.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-26T02:11:17.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T02:11:17.279 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-04-26T02:11:17.855 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-26T02:11:17.855 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-26T02:11:17.855 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-04-26T02:11:17.903 INFO:teuthology.orchestra.run.smithi087.stdout:5.15.0-105-generic 2024-04-26T02:11:17.903 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T02:11:17.903 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T02:11:17.903 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T02:11:18.903 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-26T02:11:18.904 DEBUG:teuthology.orchestra.run.smithi087:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T02:11:18.977 INFO:teuthology.orchestra.run.smithi087.stdout:ttyS1 2024-04-26T02:11:19.009 DEBUG:teuthology.parallel:result is None 2024-04-26T02:11:19.009 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T02:11:19.017 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T02:11:19.018 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T02:11:19.020 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T02:11:19.026 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T02:11:19.054 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T02:11:19.132 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T02:11:19.133 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T02:11:19.135 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T02:11:19.238 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T02:11:19.248 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T02:11:19.249 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T02:11:19.251 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T02:11:19.274 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T02:11:19.276 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T02:11:19.284 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T02:11:19.284 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T02:11:19.286 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T02:11:19.297 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T02:11:19.297 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T02:11:19.331 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T02:11:19.344 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T02:11:19.354 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T02:11:19.354 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T02:11:19.388 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T02:11:19.393 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T02:11:19.439 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T02:11:19.487 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:11:19.487 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T02:11:19.550 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T02:11:19.556 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T02:11:19.604 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:11:19.604 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T02:11:19.677 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2024-04-26T02:11:19.680 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-04-26T02:11:19.747 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T02:11:19.755 INFO:teuthology.task.internal:Starting timer... 2024-04-26T02:11:19.755 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T02:11:19.849 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T02:11:19.908 DEBUG:teuthology.task.selinux:Excluding smithi087: OS 'ubuntu' does not support SELinux 2024-04-26T02:11:19.908 DEBUG:teuthology.task.selinux:Excluding smithi122: OS 'ubuntu' does not support SELinux 2024-04-26T02:11:19.908 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T02:11:19.909 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-26T02:11:19.909 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T02:11:19.909 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T02:11:19.967 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T02:11:19.983 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T02:11:19.984 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi087.front.sepia.ceph.com,smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T02:18:24.852 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi087.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2024-04-26T02:18:24.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-26T02:18:24.854 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T02:18:24.937 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-04-26T02:18:25.018 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-26T02:18:25.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-26T02:18:25.019 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T02:18:25.100 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-04-26T02:18:25.174 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-26T02:18:25.175 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T02:18:25.191 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T02:18:25.192 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T02:18:25.192 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T02:18:25.195 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T02:18:25.196 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T02:18:25.226 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T02:18:25.226 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: Command line: ntpd -gq 2024-04-26T02:18:25.226 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: ---------------------------------------------------- 2024-04-26T02:18:25.226 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T02:18:25.226 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T02:18:25.226 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: corporation. Support and training for ntp-4 are 2024-04-26T02:18:25.226 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: available at https://www.nwtime.org/support 2024-04-26T02:18:25.226 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: ---------------------------------------------------- 2024-04-26T02:18:25.226 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: proto: precision = 0.060 usec (-24) 2024-04-26T02:18:25.227 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: basedate set to 2022-02-04 2024-04-26T02:18:25.227 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: gps base set to 2022-02-06 (week 2196) 2024-04-26T02:18:25.227 INFO:teuthology.orchestra.run.smithi087.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T02:18:25.227 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T02:18:25.227 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T02:18:25.228 INFO:teuthology.orchestra.run.smithi087.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T02:18:25.228 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T02:18:25.229 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T02:18:25.229 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T02:18:25.229 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: Listen normally on 3 enp3s0f1 172.21.15.87:123 2024-04-26T02:18:25.229 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: Listen normally on 4 lo [::1]:123 2024-04-26T02:18:25.229 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d5f5%5]:123 2024-04-26T02:18:25.229 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:25 ntpd[17247]: Listening on routing socket on fd #22 for interface updates 2024-04-26T02:18:25.242 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T02:18:25.242 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: Command line: ntpd -gq 2024-04-26T02:18:25.242 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: ---------------------------------------------------- 2024-04-26T02:18:25.242 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T02:18:25.242 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T02:18:25.242 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: corporation. Support and training for ntp-4 are 2024-04-26T02:18:25.242 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: available at https://www.nwtime.org/support 2024-04-26T02:18:25.243 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: ---------------------------------------------------- 2024-04-26T02:18:25.243 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: proto: precision = 0.045 usec (-24) 2024-04-26T02:18:25.243 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: basedate set to 2022-02-04 2024-04-26T02:18:25.243 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: gps base set to 2022-02-06 (week 2196) 2024-04-26T02:18:25.243 INFO:teuthology.orchestra.run.smithi122.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T02:18:25.244 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T02:18:25.244 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T02:18:25.244 INFO:teuthology.orchestra.run.smithi122.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T02:18:25.245 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T02:18:25.245 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T02:18:25.245 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T02:18:25.245 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: Listen normally on 3 enp3s0f1 172.21.15.122:123 2024-04-26T02:18:25.245 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: Listen normally on 4 lo [::1]:123 2024-04-26T02:18:25.245 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d515%5]:123 2024-04-26T02:18:25.245 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:25 ntpd[17235]: Listening on routing socket on fd #22 for interface updates 2024-04-26T02:18:26.225 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:26 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:26.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:26 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:26.242 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:26 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:26.242 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:26 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:26.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:26 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:26.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:26 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:26.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:26 ntpd[17235]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:26.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:26 ntpd[17235]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:27.242 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:27 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:27.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:27 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:27.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:27 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:27.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:27 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:27.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:27 ntpd[17235]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:27.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:27 ntpd[17235]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:28.225 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:28 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:28.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:28 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:28.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:28 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:28.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:28 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:28.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:28 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:28.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:28 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:28.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:28 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:28.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:28 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:28.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:28 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:28.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:28 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:28.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:28 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:28.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:28 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:28.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:28 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:28.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:28 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:28.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:28 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:28.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:28 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:28.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:28 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:28.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:28 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:28.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:28 ntpd[17235]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:28.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:28 ntpd[17235]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:29.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:29 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:29.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:29 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:29.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:29 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:29.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:29 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:29.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:29 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:29.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:29 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:29.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:29 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:29.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:29 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:29.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:29 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:29.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:29 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:29.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:29 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:29.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:29 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:29.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:29 ntpd[17235]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:29.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:29 ntpd[17235]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:29.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:29 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:29.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:29 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:29.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:29 ntpd[17235]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:29.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:29 ntpd[17235]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:30.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:30 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:30.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:30 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:30.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:30 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:30.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:30 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:30.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:30 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:30.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:30 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:30.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:30 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:30.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:30 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:30.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:30 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:30.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:30 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:30.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:30 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:30.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:30 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:30.242 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:30 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:30.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:30 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:30.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:30 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:30.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:30 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:30.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:30 ntpd[17235]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:30.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:30 ntpd[17235]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:31.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:31 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:31.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:31 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:31.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:31 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:31.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:31 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:31.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:31 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:31.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:31 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:31.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:31 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:31.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:31 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:31.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:31 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:31.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:31 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:31.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:31 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:31.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:31 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:31.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:31 ntpd[17235]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:31.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:31 ntpd[17235]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:31.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:31 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:31.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:31 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:31.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:31 ntpd[17235]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:31.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:31 ntpd[17235]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:32.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:32 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:32.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:32 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:32.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:32 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:32.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:32 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:32.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:32 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:32.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:32 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:32.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:32 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:32.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:32 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:32.228 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:32 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:32.228 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:32 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:32.228 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:32 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:32.228 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:32 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:32.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:32 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:32.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:32 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:32.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:32 ntpd[17235]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:32.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:32 ntpd[17235]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:32.243 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:32 ntpd[17235]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:32.243 INFO:teuthology.orchestra.run.smithi122.stdout:26 Apr 02:18:32 ntpd[17235]: ntpd: time slew +0.000128 s 2024-04-26T02:18:32.243 INFO:teuthology.orchestra.run.smithi122.stdout:ntpd: time slew +0.000128s 2024-04-26T02:18:32.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:32 ntpd[17235]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:32.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:32 ntpd[17235]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T02:18:32.244 INFO:teuthology.orchestra.run.smithi122.stderr:26 Apr 02:18:32 ntpd[17235]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T02:18:32.297 INFO:teuthology.orchestra.run.smithi122.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T02:18:32.298 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================== 2024-04-26T02:18:32.298 INFO:teuthology.orchestra.run.smithi122.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T02:18:32.298 INFO:teuthology.orchestra.run.smithi122.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T02:18:32.298 INFO:teuthology.orchestra.run.smithi122.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T02:18:32.298 INFO:teuthology.orchestra.run.smithi122.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T02:18:33.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:33 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:33.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:33 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:33.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:33 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:33.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:33 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:33.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:33 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:33.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:33 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:34.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:34 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:34.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:34 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:34.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:34 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:34.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:34 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:34.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:34 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T02:18:34.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:34 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T02:18:34.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:34 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T02:18:34.226 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:34 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T02:18:34.227 INFO:teuthology.orchestra.run.smithi087.stdout:26 Apr 02:18:34 ntpd[17247]: ntpd: time slew +0.000400 s 2024-04-26T02:18:34.227 INFO:teuthology.orchestra.run.smithi087.stdout:ntpd: time slew +0.000400s 2024-04-26T02:18:34.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:34 ntpd[17247]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T02:18:34.227 INFO:teuthology.orchestra.run.smithi087.stderr:26 Apr 02:18:34 ntpd[17247]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T02:18:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T02:18:34.281 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================== 2024-04-26T02:18:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T02:18:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T02:18:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T02:18:34.281 INFO:teuthology.orchestra.run.smithi087.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T02:18:34.282 INFO:teuthology.run_tasks:Running task install... 2024-04-26T02:18:34.291 DEBUG:teuthology.task.install:project ceph 2024-04-26T02:18:34.291 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'}} 2024-04-26T02:18:34.291 DEBUG:teuthology.task.install:config {'extra_system_packages': {'deb': ['sqlite3'], 'rpm': ['sqlite-devel']}, 'flavor': 'default', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'} 2024-04-26T02:18:34.291 INFO:teuthology.task.install:Using flavor: default 2024-04-26T02:18:34.296 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T02:18:34.297 INFO:teuthology.task.install:extra packages: [] 2024-04-26T02:18:34.297 DEBUG:teuthology.orchestra.run.smithi087:> sudo apt-key list | grep Ceph 2024-04-26T02:18:34.297 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-key list | grep Ceph 2024-04-26T02:18:34.363 INFO:teuthology.orchestra.run.smithi122.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T02:18:34.376 INFO:teuthology.orchestra.run.smithi087.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T02:18:34.394 INFO:teuthology.orchestra.run.smithi122.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T02:18:34.394 INFO:teuthology.orchestra.run.smithi122.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T02:18:34.394 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T02:18:34.394 INFO:teuthology.task.install.deb:Installing system (non-project) packages: sqlite3 on remote deb x86_64 2024-04-26T02:18:34.395 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T02:18:34.404 INFO:teuthology.orchestra.run.smithi087.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T02:18:34.404 INFO:teuthology.orchestra.run.smithi087.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T02:18:34.405 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T02:18:34.405 INFO:teuthology.task.install.deb:Installing system (non-project) packages: sqlite3 on remote deb x86_64 2024-04-26T02:18:34.406 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T02:18:34.545 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default/ 2024-04-26T02:18:34.545 INFO:teuthology.task.install.deb:Package version is 19.0.0-2457-gddeaa917-1jammy 2024-04-26T02:18:34.615 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default/ 2024-04-26T02:18:34.615 INFO:teuthology.task.install.deb:Package version is 19.0.0-2457-gddeaa917-1jammy 2024-04-26T02:18:34.655 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:18:34.655 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T02:18:34.670 DEBUG:teuthology.orchestra.run.smithi087:> sudo apt-get update 2024-04-26T02:18:34.729 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:18:34.729 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T02:18:34.744 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get update 2024-04-26T02:18:34.901 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T02:18:34.901 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T02:18:34.950 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T02:18:34.991 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T02:18:34.992 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T02:18:35.008 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T02:18:35.044 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T02:18:35.097 INFO:teuthology.orchestra.run.smithi087.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T02:18:35.100 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T02:18:35.165 INFO:teuthology.orchestra.run.smithi087.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T02:18:35.172 INFO:teuthology.orchestra.run.smithi122.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T02:18:35.243 INFO:teuthology.orchestra.run.smithi087.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T02:18:35.246 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T02:18:35.314 INFO:teuthology.orchestra.run.smithi087.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-26T02:18:35.317 INFO:teuthology.orchestra.run.smithi122.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T02:18:35.354 INFO:teuthology.orchestra.run.smithi122.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,421 B] 2024-04-26T02:18:35.364 INFO:teuthology.orchestra.run.smithi087.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,421 B] 2024-04-26T02:18:35.400 INFO:teuthology.orchestra.run.smithi122.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-26T02:18:35.455 INFO:teuthology.orchestra.run.smithi087.stdout:Fetched 28.9 kB in 1s (49.0 kB/s) 2024-04-26T02:18:35.486 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 28.9 kB in 1s (52.3 kB/s) 2024-04-26T02:18:36.426 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2024-04-26T02:18:36.441 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2457-gddeaa917-1jammy cephadm=19.0.0-2457-gddeaa917-1jammy ceph-mds=19.0.0-2457-gddeaa917-1jammy ceph-mgr=19.0.0-2457-gddeaa917-1jammy ceph-common=19.0.0-2457-gddeaa917-1jammy ceph-fuse=19.0.0-2457-gddeaa917-1jammy ceph-test=19.0.0-2457-gddeaa917-1jammy ceph-volume=19.0.0-2457-gddeaa917-1jammy radosgw=19.0.0-2457-gddeaa917-1jammy python3-rados=19.0.0-2457-gddeaa917-1jammy python3-rgw=19.0.0-2457-gddeaa917-1jammy python3-cephfs=19.0.0-2457-gddeaa917-1jammy python3-rbd=19.0.0-2457-gddeaa917-1jammy libcephfs2=19.0.0-2457-gddeaa917-1jammy libcephfs-dev=19.0.0-2457-gddeaa917-1jammy librados2=19.0.0-2457-gddeaa917-1jammy librbd1=19.0.0-2457-gddeaa917-1jammy rbd-fuse=19.0.0-2457-gddeaa917-1jammy 2024-04-26T02:18:36.447 INFO:teuthology.orchestra.run.smithi087.stdout:Reading package lists... 2024-04-26T02:18:36.464 DEBUG:teuthology.orchestra.run.smithi087:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2457-gddeaa917-1jammy cephadm=19.0.0-2457-gddeaa917-1jammy ceph-mds=19.0.0-2457-gddeaa917-1jammy ceph-mgr=19.0.0-2457-gddeaa917-1jammy ceph-common=19.0.0-2457-gddeaa917-1jammy ceph-fuse=19.0.0-2457-gddeaa917-1jammy ceph-test=19.0.0-2457-gddeaa917-1jammy ceph-volume=19.0.0-2457-gddeaa917-1jammy radosgw=19.0.0-2457-gddeaa917-1jammy python3-rados=19.0.0-2457-gddeaa917-1jammy python3-rgw=19.0.0-2457-gddeaa917-1jammy python3-cephfs=19.0.0-2457-gddeaa917-1jammy python3-rbd=19.0.0-2457-gddeaa917-1jammy libcephfs2=19.0.0-2457-gddeaa917-1jammy libcephfs-dev=19.0.0-2457-gddeaa917-1jammy librados2=19.0.0-2457-gddeaa917-1jammy librbd1=19.0.0-2457-gddeaa917-1jammy rbd-fuse=19.0.0-2457-gddeaa917-1jammy 2024-04-26T02:18:36.501 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2024-04-26T02:18:36.526 INFO:teuthology.orchestra.run.smithi087.stdout:Reading package lists... 2024-04-26T02:18:36.700 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2024-04-26T02:18:36.701 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2024-04-26T02:18:36.727 INFO:teuthology.orchestra.run.smithi087.stdout:Building dependency tree... 2024-04-26T02:18:36.728 INFO:teuthology.orchestra.run.smithi087.stdout:Reading state information... 2024-04-26T02:18:36.892 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T02:18:36.892 INFO:teuthology.orchestra.run.smithi122.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T02:18:36.892 INFO:teuthology.orchestra.run.smithi122.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T02:18:36.892 INFO:teuthology.orchestra.run.smithi122.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T02:18:36.892 INFO:teuthology.orchestra.run.smithi122.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T02:18:36.894 INFO:teuthology.orchestra.run.smithi122.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T02:18:36.894 INFO:teuthology.orchestra.run.smithi122.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T02:18:36.894 INFO:teuthology.orchestra.run.smithi122.stdout:The following additional packages will be installed: 2024-04-26T02:18:36.894 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T02:18:36.894 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T02:18:36.894 INFO:teuthology.orchestra.run.smithi122.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T02:18:36.894 INFO:teuthology.orchestra.run.smithi122.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T02:18:36.895 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T02:18:36.895 INFO:teuthology.orchestra.run.smithi122.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T02:18:36.895 INFO:teuthology.orchestra.run.smithi122.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T02:18:36.895 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T02:18:36.895 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T02:18:36.895 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T02:18:36.895 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T02:18:36.895 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T02:18:36.895 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T02:18:36.895 INFO:teuthology.orchestra.run.smithi122.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T02:18:36.895 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T02:18:36.895 INFO:teuthology.orchestra.run.smithi122.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T02:18:36.895 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T02:18:36.896 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T02:18:36.896 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T02:18:36.896 INFO:teuthology.orchestra.run.smithi122.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T02:18:36.896 INFO:teuthology.orchestra.run.smithi122.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T02:18:36.896 INFO:teuthology.orchestra.run.smithi122.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T02:18:36.896 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T02:18:36.896 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet zip 2024-04-26T02:18:36.896 INFO:teuthology.orchestra.run.smithi122.stdout:Suggested packages: 2024-04-26T02:18:36.896 INFO:teuthology.orchestra.run.smithi122.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T02:18:36.896 INFO:teuthology.orchestra.run.smithi122.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T02:18:36.896 INFO:teuthology.orchestra.run.smithi122.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T02:18:36.897 INFO:teuthology.orchestra.run.smithi122.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T02:18:36.897 INFO:teuthology.orchestra.run.smithi122.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T02:18:36.897 INFO:teuthology.orchestra.run.smithi122.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T02:18:36.897 INFO:teuthology.orchestra.run.smithi122.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T02:18:36.897 INFO:teuthology.orchestra.run.smithi122.stdout:Recommended packages: 2024-04-26T02:18:36.897 INFO:teuthology.orchestra.run.smithi122.stdout: btrfs-tools 2024-04-26T02:18:36.921 INFO:teuthology.orchestra.run.smithi087.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T02:18:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T02:18:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T02:18:36.921 INFO:teuthology.orchestra.run.smithi087.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T02:18:36.922 INFO:teuthology.orchestra.run.smithi087.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T02:18:36.922 INFO:teuthology.orchestra.run.smithi087.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T02:18:36.922 INFO:teuthology.orchestra.run.smithi087.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T02:18:36.922 INFO:teuthology.orchestra.run.smithi087.stdout:The following additional packages will be installed: 2024-04-26T02:18:36.922 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T02:18:36.922 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T02:18:36.924 INFO:teuthology.orchestra.run.smithi087.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T02:18:36.924 INFO:teuthology.orchestra.run.smithi087.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T02:18:36.924 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T02:18:36.924 INFO:teuthology.orchestra.run.smithi087.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T02:18:36.924 INFO:teuthology.orchestra.run.smithi087.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T02:18:36.924 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T02:18:36.924 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T02:18:36.924 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T02:18:36.924 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T02:18:36.924 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T02:18:36.924 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T02:18:36.925 INFO:teuthology.orchestra.run.smithi087.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T02:18:36.925 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T02:18:36.925 INFO:teuthology.orchestra.run.smithi087.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T02:18:36.925 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T02:18:36.925 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T02:18:36.925 INFO:teuthology.orchestra.run.smithi087.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T02:18:36.925 INFO:teuthology.orchestra.run.smithi087.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T02:18:36.925 INFO:teuthology.orchestra.run.smithi087.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T02:18:36.925 INFO:teuthology.orchestra.run.smithi087.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T02:18:36.925 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T02:18:36.925 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet zip 2024-04-26T02:18:36.926 INFO:teuthology.orchestra.run.smithi087.stdout:Suggested packages: 2024-04-26T02:18:36.926 INFO:teuthology.orchestra.run.smithi087.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T02:18:36.926 INFO:teuthology.orchestra.run.smithi087.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T02:18:36.926 INFO:teuthology.orchestra.run.smithi087.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T02:18:36.926 INFO:teuthology.orchestra.run.smithi087.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T02:18:36.926 INFO:teuthology.orchestra.run.smithi087.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T02:18:36.926 INFO:teuthology.orchestra.run.smithi087.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T02:18:36.926 INFO:teuthology.orchestra.run.smithi087.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T02:18:36.926 INFO:teuthology.orchestra.run.smithi087.stdout:Recommended packages: 2024-04-26T02:18:36.926 INFO:teuthology.orchestra.run.smithi087.stdout: btrfs-tools 2024-04-26T02:18:36.995 INFO:teuthology.orchestra.run.smithi122.stdout:The following NEW packages will be installed: 2024-04-26T02:18:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T02:18:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T02:18:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T02:18:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T02:18:36.996 INFO:teuthology.orchestra.run.smithi122.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T02:18:36.996 INFO:teuthology.orchestra.run.smithi122.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T02:18:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T02:18:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T02:18:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T02:18:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T02:18:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T02:18:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T02:18:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T02:18:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T02:18:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T02:18:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T02:18:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T02:18:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T02:18:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T02:18:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T02:18:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T02:18:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T02:18:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T02:18:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T02:18:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T02:18:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T02:18:37.030 INFO:teuthology.orchestra.run.smithi087.stdout:The following NEW packages will be installed: 2024-04-26T02:18:37.030 INFO:teuthology.orchestra.run.smithi087.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T02:18:37.030 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T02:18:37.030 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T02:18:37.030 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T02:18:37.030 INFO:teuthology.orchestra.run.smithi087.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T02:18:37.031 INFO:teuthology.orchestra.run.smithi087.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T02:18:37.032 INFO:teuthology.orchestra.run.smithi087.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T02:18:37.032 INFO:teuthology.orchestra.run.smithi087.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T02:18:37.032 INFO:teuthology.orchestra.run.smithi087.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T02:18:37.032 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T02:18:37.032 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T02:18:37.032 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T02:18:37.032 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T02:18:37.032 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T02:18:37.032 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T02:18:37.032 INFO:teuthology.orchestra.run.smithi087.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T02:18:37.032 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T02:18:37.032 INFO:teuthology.orchestra.run.smithi087.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T02:18:37.032 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T02:18:37.033 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T02:18:37.033 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T02:18:37.033 INFO:teuthology.orchestra.run.smithi087.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T02:18:37.033 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T02:18:37.033 INFO:teuthology.orchestra.run.smithi087.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T02:18:37.033 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T02:18:37.033 INFO:teuthology.orchestra.run.smithi087.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T02:18:37.062 INFO:teuthology.orchestra.run.smithi122.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T02:18:37.062 INFO:teuthology.orchestra.run.smithi122.stdout:Need to get 179 MB of archives. 2024-04-26T02:18:37.062 INFO:teuthology.orchestra.run.smithi122.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-26T02:18:37.062 INFO:teuthology.orchestra.run.smithi122.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T02:18:37.095 INFO:teuthology.orchestra.run.smithi087.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T02:18:37.095 INFO:teuthology.orchestra.run.smithi087.stdout:Need to get 179 MB of archives. 2024-04-26T02:18:37.095 INFO:teuthology.orchestra.run.smithi087.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-26T02:18:37.095 INFO:teuthology.orchestra.run.smithi087.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T02:18:37.169 INFO:teuthology.orchestra.run.smithi122.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T02:18:37.169 INFO:teuthology.orchestra.run.smithi087.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2457-gddeaa917-1jammy [3,545 kB] 2024-04-26T02:18:37.172 INFO:teuthology.orchestra.run.smithi122.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T02:18:37.192 INFO:teuthology.orchestra.run.smithi122.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T02:18:37.205 INFO:teuthology.orchestra.run.smithi087.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T02:18:37.209 INFO:teuthology.orchestra.run.smithi087.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T02:18:37.217 INFO:teuthology.orchestra.run.smithi122.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2457-gddeaa917-1jammy [3,545 kB] 2024-04-26T02:18:37.243 INFO:teuthology.orchestra.run.smithi087.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T02:18:37.375 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T02:18:37.384 INFO:teuthology.orchestra.run.smithi122.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T02:18:37.385 INFO:teuthology.orchestra.run.smithi087.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2457-gddeaa917-1jammy [3,218 kB] 2024-04-26T02:18:37.403 INFO:teuthology.orchestra.run.smithi122.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T02:18:37.409 INFO:teuthology.orchestra.run.smithi122.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T02:18:37.410 INFO:teuthology.orchestra.run.smithi122.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T02:18:37.412 INFO:teuthology.orchestra.run.smithi122.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T02:18:37.425 INFO:teuthology.orchestra.run.smithi122.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T02:18:37.430 INFO:teuthology.orchestra.run.smithi122.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T02:18:37.431 INFO:teuthology.orchestra.run.smithi122.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T02:18:37.433 INFO:teuthology.orchestra.run.smithi122.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T02:18:37.451 INFO:teuthology.orchestra.run.smithi122.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T02:18:37.455 INFO:teuthology.orchestra.run.smithi122.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T02:18:37.460 INFO:teuthology.orchestra.run.smithi122.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T02:18:37.484 INFO:teuthology.orchestra.run.smithi122.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T02:18:37.485 INFO:teuthology.orchestra.run.smithi122.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T02:18:37.486 INFO:teuthology.orchestra.run.smithi122.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T02:18:37.486 INFO:teuthology.orchestra.run.smithi087.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2457-gddeaa917-1jammy [711 kB] 2024-04-26T02:18:37.487 INFO:teuthology.orchestra.run.smithi122.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T02:18:37.488 INFO:teuthology.orchestra.run.smithi122.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T02:18:37.489 INFO:teuthology.orchestra.run.smithi122.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T02:18:37.489 INFO:teuthology.orchestra.run.smithi122.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T02:18:37.511 INFO:teuthology.orchestra.run.smithi122.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T02:18:37.512 INFO:teuthology.orchestra.run.smithi087.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2457-gddeaa917-1jammy [356 kB] 2024-04-26T02:18:37.512 INFO:teuthology.orchestra.run.smithi122.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T02:18:37.512 INFO:teuthology.orchestra.run.smithi122.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T02:18:37.523 INFO:teuthology.orchestra.run.smithi087.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2457-gddeaa917-1jammy [32.8 kB] 2024-04-26T02:18:37.524 INFO:teuthology.orchestra.run.smithi087.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2457-gddeaa917-1jammy [180 kB] 2024-04-26T02:18:37.530 INFO:teuthology.orchestra.run.smithi087.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T02:18:37.530 INFO:teuthology.orchestra.run.smithi087.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2457-gddeaa917-1jammy [65.8 kB] 2024-04-26T02:18:37.532 INFO:teuthology.orchestra.run.smithi087.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2457-gddeaa917-1jammy [330 kB] 2024-04-26T02:18:37.543 INFO:teuthology.orchestra.run.smithi087.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2457-gddeaa917-1jammy [6,775 kB] 2024-04-26T02:18:37.548 INFO:teuthology.orchestra.run.smithi122.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T02:18:37.549 INFO:teuthology.orchestra.run.smithi122.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T02:18:37.549 INFO:teuthology.orchestra.run.smithi122.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T02:18:37.550 INFO:teuthology.orchestra.run.smithi122.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T02:18:37.551 INFO:teuthology.orchestra.run.smithi122.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T02:18:37.552 INFO:teuthology.orchestra.run.smithi122.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T02:18:37.552 INFO:teuthology.orchestra.run.smithi122.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T02:18:37.566 INFO:teuthology.orchestra.run.smithi087.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T02:18:37.578 INFO:teuthology.orchestra.run.smithi122.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T02:18:37.579 INFO:teuthology.orchestra.run.smithi122.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T02:18:37.580 INFO:teuthology.orchestra.run.smithi122.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T02:18:37.599 INFO:teuthology.orchestra.run.smithi122.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T02:18:37.599 INFO:teuthology.orchestra.run.smithi122.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T02:18:37.600 INFO:teuthology.orchestra.run.smithi122.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T02:18:37.601 INFO:teuthology.orchestra.run.smithi122.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T02:18:37.604 INFO:teuthology.orchestra.run.smithi122.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T02:18:37.608 INFO:teuthology.orchestra.run.smithi122.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T02:18:37.609 INFO:teuthology.orchestra.run.smithi122.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T02:18:37.619 INFO:teuthology.orchestra.run.smithi122.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T02:18:37.621 INFO:teuthology.orchestra.run.smithi122.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T02:18:37.625 INFO:teuthology.orchestra.run.smithi087.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T02:18:37.625 INFO:teuthology.orchestra.run.smithi122.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T02:18:37.640 INFO:teuthology.orchestra.run.smithi122.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T02:18:37.641 INFO:teuthology.orchestra.run.smithi122.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T02:18:37.645 INFO:teuthology.orchestra.run.smithi122.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T02:18:37.648 INFO:teuthology.orchestra.run.smithi087.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T02:18:37.650 INFO:teuthology.orchestra.run.smithi087.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T02:18:37.653 INFO:teuthology.orchestra.run.smithi087.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T02:18:37.729 INFO:teuthology.orchestra.run.smithi122.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T02:18:37.730 INFO:teuthology.orchestra.run.smithi087.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T02:18:37.731 INFO:teuthology.orchestra.run.smithi122.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T02:18:37.733 INFO:teuthology.orchestra.run.smithi122.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T02:18:37.733 INFO:teuthology.orchestra.run.smithi122.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T02:18:37.736 INFO:teuthology.orchestra.run.smithi122.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T02:18:37.746 INFO:teuthology.orchestra.run.smithi087.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T02:18:37.762 INFO:teuthology.orchestra.run.smithi087.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T02:18:37.764 INFO:teuthology.orchestra.run.smithi122.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T02:18:37.767 INFO:teuthology.orchestra.run.smithi122.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T02:18:37.767 INFO:teuthology.orchestra.run.smithi122.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T02:18:37.769 INFO:teuthology.orchestra.run.smithi087.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T02:18:37.770 INFO:teuthology.orchestra.run.smithi087.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T02:18:37.777 INFO:teuthology.orchestra.run.smithi087.stdout:Get:25 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2457-gddeaa917-1jammy [112 kB] 2024-04-26T02:18:37.780 INFO:teuthology.orchestra.run.smithi087.stdout:Get:26 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2457-gddeaa917-1jammy [466 kB] 2024-04-26T02:18:37.785 INFO:teuthology.orchestra.run.smithi087.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T02:18:37.795 INFO:teuthology.orchestra.run.smithi087.stdout:Get:28 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2457-gddeaa917-1jammy [25.6 MB] 2024-04-26T02:18:37.801 INFO:teuthology.orchestra.run.smithi122.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T02:18:37.801 INFO:teuthology.orchestra.run.smithi122.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T02:18:37.801 INFO:teuthology.orchestra.run.smithi087.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T02:18:37.807 INFO:teuthology.orchestra.run.smithi087.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T02:18:37.808 INFO:teuthology.orchestra.run.smithi087.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T02:18:37.809 INFO:teuthology.orchestra.run.smithi087.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T02:18:37.812 INFO:teuthology.orchestra.run.smithi122.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T02:18:37.813 INFO:teuthology.orchestra.run.smithi122.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T02:18:37.813 INFO:teuthology.orchestra.run.smithi122.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T02:18:37.814 INFO:teuthology.orchestra.run.smithi122.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T02:18:37.815 INFO:teuthology.orchestra.run.smithi122.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T02:18:37.815 INFO:teuthology.orchestra.run.smithi122.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T02:18:37.820 INFO:teuthology.orchestra.run.smithi122.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T02:18:37.821 INFO:teuthology.orchestra.run.smithi087.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T02:18:37.822 INFO:teuthology.orchestra.run.smithi087.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T02:18:37.823 INFO:teuthology.orchestra.run.smithi087.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T02:18:37.823 INFO:teuthology.orchestra.run.smithi087.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T02:18:37.824 INFO:teuthology.orchestra.run.smithi087.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T02:18:37.833 INFO:teuthology.orchestra.run.smithi122.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T02:18:37.835 INFO:teuthology.orchestra.run.smithi122.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T02:18:37.836 INFO:teuthology.orchestra.run.smithi122.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T02:18:37.842 INFO:teuthology.orchestra.run.smithi087.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T02:18:37.846 INFO:teuthology.orchestra.run.smithi087.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T02:18:37.854 INFO:teuthology.orchestra.run.smithi122.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T02:18:37.857 INFO:teuthology.orchestra.run.smithi122.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T02:18:37.858 INFO:teuthology.orchestra.run.smithi122.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T02:18:37.859 INFO:teuthology.orchestra.run.smithi122.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T02:18:37.860 INFO:teuthology.orchestra.run.smithi087.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T02:18:37.865 INFO:teuthology.orchestra.run.smithi087.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T02:18:37.865 INFO:teuthology.orchestra.run.smithi122.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T02:18:37.866 INFO:teuthology.orchestra.run.smithi122.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T02:18:37.866 INFO:teuthology.orchestra.run.smithi087.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T02:18:37.868 INFO:teuthology.orchestra.run.smithi087.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T02:18:37.871 INFO:teuthology.orchestra.run.smithi122.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T02:18:37.872 INFO:teuthology.orchestra.run.smithi087.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T02:18:37.873 INFO:teuthology.orchestra.run.smithi087.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T02:18:37.873 INFO:teuthology.orchestra.run.smithi087.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T02:18:37.875 INFO:teuthology.orchestra.run.smithi087.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T02:18:37.875 INFO:teuthology.orchestra.run.smithi122.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T02:18:37.876 INFO:teuthology.orchestra.run.smithi122.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T02:18:37.876 INFO:teuthology.orchestra.run.smithi122.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T02:18:37.891 INFO:teuthology.orchestra.run.smithi087.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T02:18:37.891 INFO:teuthology.orchestra.run.smithi087.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T02:18:37.898 INFO:teuthology.orchestra.run.smithi122.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T02:18:37.900 INFO:teuthology.orchestra.run.smithi122.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T02:18:37.911 INFO:teuthology.orchestra.run.smithi122.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T02:18:37.912 INFO:teuthology.orchestra.run.smithi122.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T02:18:37.912 INFO:teuthology.orchestra.run.smithi122.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T02:18:37.916 INFO:teuthology.orchestra.run.smithi122.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T02:18:37.917 INFO:teuthology.orchestra.run.smithi122.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T02:18:37.957 INFO:teuthology.orchestra.run.smithi087.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T02:18:37.958 INFO:teuthology.orchestra.run.smithi087.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T02:18:37.963 INFO:teuthology.orchestra.run.smithi087.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T02:18:37.967 INFO:teuthology.orchestra.run.smithi087.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T02:18:38.011 INFO:teuthology.orchestra.run.smithi087.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T02:18:38.015 INFO:teuthology.orchestra.run.smithi087.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T02:18:38.016 INFO:teuthology.orchestra.run.smithi087.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T02:18:38.019 INFO:teuthology.orchestra.run.smithi087.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T02:18:38.021 INFO:teuthology.orchestra.run.smithi087.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T02:18:38.025 INFO:teuthology.orchestra.run.smithi087.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T02:18:38.026 INFO:teuthology.orchestra.run.smithi087.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T02:18:38.028 INFO:teuthology.orchestra.run.smithi087.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T02:18:38.041 INFO:teuthology.orchestra.run.smithi087.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T02:18:38.496 INFO:teuthology.orchestra.run.smithi087.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T02:18:38.498 INFO:teuthology.orchestra.run.smithi087.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T02:18:38.501 INFO:teuthology.orchestra.run.smithi087.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T02:18:38.502 INFO:teuthology.orchestra.run.smithi087.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T02:18:38.504 INFO:teuthology.orchestra.run.smithi087.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T02:18:38.573 INFO:teuthology.orchestra.run.smithi087.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T02:18:38.585 INFO:teuthology.orchestra.run.smithi087.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T02:18:38.585 INFO:teuthology.orchestra.run.smithi087.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T02:18:38.709 INFO:teuthology.orchestra.run.smithi087.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T02:18:38.710 INFO:teuthology.orchestra.run.smithi087.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T02:18:38.747 INFO:teuthology.orchestra.run.smithi087.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T02:18:38.748 INFO:teuthology.orchestra.run.smithi087.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T02:18:38.749 INFO:teuthology.orchestra.run.smithi087.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T02:18:38.812 INFO:teuthology.orchestra.run.smithi087.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T02:18:38.812 INFO:teuthology.orchestra.run.smithi087.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2457-gddeaa917-1jammy [5,084 kB] 2024-04-26T02:18:38.813 INFO:teuthology.orchestra.run.smithi087.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T02:18:38.814 INFO:teuthology.orchestra.run.smithi087.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T02:18:38.823 INFO:teuthology.orchestra.run.smithi087.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T02:18:38.827 INFO:teuthology.orchestra.run.smithi087.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T02:18:38.830 INFO:teuthology.orchestra.run.smithi087.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T02:18:38.832 INFO:teuthology.orchestra.run.smithi087.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T02:18:38.840 INFO:teuthology.orchestra.run.smithi087.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T02:18:38.849 INFO:teuthology.orchestra.run.smithi087.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T02:18:38.854 INFO:teuthology.orchestra.run.smithi087.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T02:18:38.856 INFO:teuthology.orchestra.run.smithi087.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T02:18:38.887 INFO:teuthology.orchestra.run.smithi087.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T02:18:38.890 INFO:teuthology.orchestra.run.smithi087.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T02:18:38.891 INFO:teuthology.orchestra.run.smithi087.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2457-gddeaa917-1jammy [244 kB] 2024-04-26T02:18:38.894 INFO:teuthology.orchestra.run.smithi087.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2457-gddeaa917-1jammy [124 kB] 2024-04-26T02:18:38.895 INFO:teuthology.orchestra.run.smithi087.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2457-gddeaa917-1jammy [1,527 kB] 2024-04-26T02:18:38.909 INFO:teuthology.orchestra.run.smithi087.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T02:18:38.910 INFO:teuthology.orchestra.run.smithi087.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T02:18:38.912 INFO:teuthology.orchestra.run.smithi087.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T02:18:38.913 INFO:teuthology.orchestra.run.smithi087.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T02:18:38.922 INFO:teuthology.orchestra.run.smithi087.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T02:18:38.926 INFO:teuthology.orchestra.run.smithi087.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T02:18:38.928 INFO:teuthology.orchestra.run.smithi087.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2457-gddeaa917-1jammy [6,245 kB] 2024-04-26T02:18:38.948 INFO:teuthology.orchestra.run.smithi087.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T02:18:38.948 INFO:teuthology.orchestra.run.smithi087.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T02:18:38.949 INFO:teuthology.orchestra.run.smithi087.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T02:18:38.956 INFO:teuthology.orchestra.run.smithi087.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T02:18:38.959 INFO:teuthology.orchestra.run.smithi087.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T02:18:39.143 INFO:teuthology.orchestra.run.smithi087.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2457-gddeaa917-1jammy [22.7 MB] 2024-04-26T02:18:40.024 INFO:teuthology.orchestra.run.smithi087.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2457-gddeaa917-1jammy [14.1 kB] 2024-04-26T02:18:40.024 INFO:teuthology.orchestra.run.smithi087.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2457-gddeaa917-1jammy [869 kB] 2024-04-26T02:18:40.041 INFO:teuthology.orchestra.run.smithi087.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2457-gddeaa917-1jammy [2,438 kB] 2024-04-26T02:18:40.062 INFO:teuthology.orchestra.run.smithi122.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2457-gddeaa917-1jammy [3,218 kB] 2024-04-26T02:18:40.083 INFO:teuthology.orchestra.run.smithi087.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2457-gddeaa917-1jammy [791 kB] 2024-04-26T02:18:40.092 INFO:teuthology.orchestra.run.smithi087.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2457-gddeaa917-1jammy [147 kB] 2024-04-26T02:18:40.094 INFO:teuthology.orchestra.run.smithi087.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2457-gddeaa917-1jammy [2,293 kB] 2024-04-26T02:18:40.151 INFO:teuthology.orchestra.run.smithi087.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2457-gddeaa917-1jammy [8,625 kB] 2024-04-26T02:18:40.447 INFO:teuthology.orchestra.run.smithi087.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2457-gddeaa917-1jammy [14.2 kB] 2024-04-26T02:18:40.448 INFO:teuthology.orchestra.run.smithi087.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2457-gddeaa917-1jammy [50.7 MB] 2024-04-26T02:18:42.234 INFO:teuthology.orchestra.run.smithi122.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2457-gddeaa917-1jammy [711 kB] 2024-04-26T02:18:42.453 INFO:teuthology.orchestra.run.smithi087.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2457-gddeaa917-1jammy [121 kB] 2024-04-26T02:18:42.455 INFO:teuthology.orchestra.run.smithi087.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2457-gddeaa917-1jammy [39.0 kB] 2024-04-26T02:18:42.456 INFO:teuthology.orchestra.run.smithi087.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2457-gddeaa917-1jammy [13.4 MB] 2024-04-26T02:18:42.564 INFO:teuthology.orchestra.run.smithi122.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2457-gddeaa917-1jammy [356 kB] 2024-04-26T02:18:42.838 INFO:teuthology.orchestra.run.smithi122.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2457-gddeaa917-1jammy [32.8 kB] 2024-04-26T02:18:42.846 INFO:teuthology.orchestra.run.smithi122.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2457-gddeaa917-1jammy [180 kB] 2024-04-26T02:18:42.878 INFO:teuthology.orchestra.run.smithi087.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2457-gddeaa917-1jammy [91.2 kB] 2024-04-26T02:18:42.900 INFO:teuthology.orchestra.run.smithi122.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2457-gddeaa917-1jammy [65.8 kB] 2024-04-26T02:18:42.979 INFO:teuthology.orchestra.run.smithi122.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2457-gddeaa917-1jammy [330 kB] 2024-04-26T02:18:43.194 INFO:teuthology.orchestra.run.smithi122.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2457-gddeaa917-1jammy [6,775 kB] 2024-04-26T02:18:43.438 INFO:teuthology.orchestra.run.smithi087.stdout:Fetched 179 MB in 6s (30.7 MB/s) 2024-04-26T02:18:43.589 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T02:18:43.661 INFO:teuthology.orchestra.run.smithi087.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T02:18:43.665 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T02:18:43.739 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T02:18:44.275 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T02:18:44.302 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T02:18:44.342 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T02:18:44.687 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T02:18:44.713 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T02:18:44.753 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T02:18:44.897 INFO:teuthology.orchestra.run.smithi122.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2457-gddeaa917-1jammy [112 kB] 2024-04-26T02:18:44.900 INFO:teuthology.orchestra.run.smithi122.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2457-gddeaa917-1jammy [466 kB] 2024-04-26T02:18:45.015 INFO:teuthology.orchestra.run.smithi122.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2457-gddeaa917-1jammy [25.6 MB] 2024-04-26T02:18:45.282 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T02:18:45.308 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T02:18:45.365 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:18:45.785 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T02:18:45.811 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T02:18:45.851 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:18:46.205 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T02:18:46.231 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T02:18:46.271 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:18:46.666 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T02:18:46.692 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T02:18:46.741 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T02:18:47.144 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package librados2. 2024-04-26T02:18:47.170 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../007-librados2_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:18:47.219 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking librados2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:18:47.663 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package librbd1. 2024-04-26T02:18:47.689 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../008-librbd1_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:18:47.730 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking librbd1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:18:48.216 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T02:18:48.242 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:18:48.291 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libcephfs2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:18:48.602 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-rados. 2024-04-26T02:18:48.628 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:18:48.677 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-rados (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:18:49.038 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T02:18:49.064 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:18:49.104 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-ceph-argparse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:18:49.357 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T02:18:49.383 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:18:49.424 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-cephfs (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:18:49.768 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T02:18:49.794 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:18:49.843 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:18:50.229 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T02:18:50.256 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T02:18:50.304 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T02:18:50.691 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T02:18:50.717 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T02:18:50.757 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T02:18:50.939 INFO:teuthology.orchestra.run.smithi122.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2457-gddeaa917-1jammy [5,084 kB] 2024-04-26T02:18:51.036 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T02:18:51.061 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:18:51.101 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-rbd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:18:51.479 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T02:18:51.505 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T02:18:51.554 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T02:18:51.636 INFO:teuthology.orchestra.run.smithi122.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2457-gddeaa917-1jammy [244 kB] 2024-04-26T02:18:51.678 INFO:teuthology.orchestra.run.smithi122.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2457-gddeaa917-1jammy [124 kB] 2024-04-26T02:18:51.690 INFO:teuthology.orchestra.run.smithi122.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2457-gddeaa917-1jammy [1,527 kB] 2024-04-26T02:18:51.873 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T02:18:51.899 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T02:18:51.930 INFO:teuthology.orchestra.run.smithi122.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2457-gddeaa917-1jammy [6,245 kB] 2024-04-26T02:18:51.957 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T02:18:52.310 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T02:18:52.336 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T02:18:52.376 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T02:18:52.713 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package lua5.1. 2024-04-26T02:18:52.724 INFO:teuthology.orchestra.run.smithi122.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2457-gddeaa917-1jammy [22.7 MB] 2024-04-26T02:18:52.739 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T02:18:52.779 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T02:18:53.099 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package lua-any. 2024-04-26T02:18:53.125 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T02:18:53.166 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T02:18:53.661 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package zip. 2024-04-26T02:18:53.677 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T02:18:53.718 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T02:18:54.038 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package unzip. 2024-04-26T02:18:54.065 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T02:18:54.105 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T02:18:54.476 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package luarocks. 2024-04-26T02:18:54.502 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T02:18:54.542 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T02:18:54.942 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package librgw2. 2024-04-26T02:18:54.968 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../025-librgw2_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:18:55.008 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking librgw2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:18:55.115 INFO:teuthology.orchestra.run.smithi122.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2457-gddeaa917-1jammy [14.1 kB] 2024-04-26T02:18:55.115 INFO:teuthology.orchestra.run.smithi122.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2457-gddeaa917-1jammy [869 kB] 2024-04-26T02:18:55.132 INFO:teuthology.orchestra.run.smithi122.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2457-gddeaa917-1jammy [2,438 kB] 2024-04-26T02:18:55.174 INFO:teuthology.orchestra.run.smithi122.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2457-gddeaa917-1jammy [791 kB] 2024-04-26T02:18:55.200 INFO:teuthology.orchestra.run.smithi122.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2457-gddeaa917-1jammy [147 kB] 2024-04-26T02:18:55.204 INFO:teuthology.orchestra.run.smithi122.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2457-gddeaa917-1jammy [2,293 kB] 2024-04-26T02:18:55.329 INFO:teuthology.orchestra.run.smithi122.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2457-gddeaa917-1jammy [8,625 kB] 2024-04-26T02:18:55.544 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T02:18:55.570 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:18:55.610 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-rgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:18:55.751 INFO:teuthology.orchestra.run.smithi122.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2457-gddeaa917-1jammy [14.2 kB] 2024-04-26T02:18:55.751 INFO:teuthology.orchestra.run.smithi122.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2457-gddeaa917-1jammy [50.7 MB] 2024-04-26T02:18:55.938 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T02:18:55.965 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T02:18:56.005 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T02:18:56.325 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T02:18:56.351 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:18:56.391 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libradosstriper1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:18:56.819 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ceph-common. 2024-04-26T02:18:56.846 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:18:56.885 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:18:57.771 INFO:teuthology.orchestra.run.smithi122.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2457-gddeaa917-1jammy [121 kB] 2024-04-26T02:18:57.773 INFO:teuthology.orchestra.run.smithi122.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2457-gddeaa917-1jammy [39.0 kB] 2024-04-26T02:18:57.774 INFO:teuthology.orchestra.run.smithi122.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2457-gddeaa917-1jammy [13.4 MB] 2024-04-26T02:18:58.128 INFO:teuthology.orchestra.run.smithi122.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2457-gddeaa917-1jammy [91.2 kB] 2024-04-26T02:18:58.224 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ceph-base. 2024-04-26T02:18:58.251 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:18:58.309 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ceph-base (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:18:58.686 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 179 MB in 21s (8,496 kB/s) 2024-04-26T02:18:58.845 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T02:18:58.929 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T02:18:58.945 INFO:teuthology.orchestra.run.smithi122.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T02:18:58.950 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T02:18:58.956 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T02:18:58.995 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T02:18:59.069 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T02:18:59.293 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T02:18:59.317 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T02:18:59.357 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T02:18:59.357 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T02:18:59.382 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T02:18:59.414 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T02:18:59.651 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T02:18:59.678 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T02:18:59.709 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T02:18:59.743 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T02:18:59.769 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T02:18:59.809 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T02:18:59.996 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T02:19:00.022 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T02:19:00.062 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T02:19:00.263 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T02:19:00.290 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T02:19:00.338 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:19:00.365 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T02:19:00.391 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T02:19:00.431 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T02:19:00.725 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T02:19:00.751 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T02:19:00.776 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T02:19:00.800 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:19:00.802 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T02:19:00.851 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T02:19:01.145 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T02:19:01.172 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T02:19:01.212 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:19:01.220 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-portend. 2024-04-26T02:19:01.247 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T02:19:01.296 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T02:19:01.540 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T02:19:01.563 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T02:19:01.599 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T02:19:01.623 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T02:19:01.650 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T02:19:01.690 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T02:19:01.927 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package librados2. 2024-04-26T02:19:01.953 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../007-librados2_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:01.993 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T02:19:01.993 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librados2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:02.019 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T02:19:02.059 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T02:19:02.413 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T02:19:02.440 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T02:19:02.505 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T02:19:02.513 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package librbd1. 2024-04-26T02:19:02.540 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../008-librbd1_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:02.580 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librbd1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:02.800 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T02:19:02.826 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T02:19:02.874 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T02:19:03.008 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T02:19:03.031 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:03.066 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libcephfs2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:03.136 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T02:19:03.162 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T02:19:03.194 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T02:19:03.353 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rados. 2024-04-26T02:19:03.379 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:03.419 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rados (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:03.463 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-mako. 2024-04-26T02:19:03.490 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T02:19:03.530 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T02:19:03.889 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T02:19:03.915 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:19:03.956 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-ceph-argparse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:04.008 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T02:19:04.034 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T02:19:04.075 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T02:19:04.175 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T02:19:04.190 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:04.225 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cephfs (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:04.377 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T02:19:04.404 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T02:19:04.445 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T02:19:04.513 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T02:19:04.538 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:19:04.578 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:04.713 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-webob. 2024-04-26T02:19:04.740 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T02:19:04.772 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T02:19:04.907 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T02:19:04.934 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T02:19:04.974 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T02:19:05.091 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T02:19:05.118 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T02:19:05.161 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T02:19:05.260 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T02:19:05.291 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T02:19:05.318 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T02:19:05.453 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T02:19:05.479 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T02:19:05.519 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T02:19:05.580 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T02:19:05.606 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:05.646 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rbd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:05.805 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-paste. 2024-04-26T02:19:05.832 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T02:19:05.872 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T02:19:06.008 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T02:19:06.034 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T02:19:06.074 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T02:19:06.175 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T02:19:06.190 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T02:19:06.233 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T02:19:06.353 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T02:19:06.368 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T02:19:06.403 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T02:19:06.570 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T02:19:06.596 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T02:19:06.636 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T02:19:06.689 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T02:19:06.716 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T02:19:06.756 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T02:19:06.939 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T02:19:06.965 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T02:19:07.005 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T02:19:07.101 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package lua5.1. 2024-04-26T02:19:07.128 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T02:19:07.168 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T02:19:07.283 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T02:19:07.299 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T02:19:07.335 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T02:19:07.480 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package lua-any. 2024-04-26T02:19:07.506 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T02:19:07.546 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T02:19:07.682 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T02:19:07.708 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T02:19:07.748 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T02:19:07.749 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package zip. 2024-04-26T02:19:07.764 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T02:19:07.799 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T02:19:08.068 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-idna. 2024-04-26T02:19:08.094 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T02:19:08.134 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T02:19:08.454 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T02:19:08.481 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T02:19:08.521 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T02:19:08.849 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-requests. 2024-04-26T02:19:08.875 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T02:19:08.887 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package unzip. 2024-04-26T02:19:08.914 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T02:19:08.915 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T02:19:09.012 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T02:19:09.227 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T02:19:09.253 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T02:19:09.285 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T02:19:09.399 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package luarocks. 2024-04-26T02:19:09.426 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T02:19:09.466 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T02:19:09.563 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T02:19:09.579 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:19:09.621 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:09.784 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package librgw2. 2024-04-26T02:19:09.810 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../025-librgw2_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:09.849 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librgw2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:09.984 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T02:19:10.011 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:10.042 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:10.419 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T02:19:10.446 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:10.486 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:10.814 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T02:19:10.841 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T02:19:10.880 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T02:19:11.146 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T02:19:11.167 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T02:19:11.172 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T02:19:11.193 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:11.212 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T02:19:11.233 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libradosstriper1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:11.507 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T02:19:11.533 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T02:19:11.573 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T02:19:11.671 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-common. 2024-04-26T02:19:11.698 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:11.737 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:11.918 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T02:19:11.944 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:11.984 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ceph-mgr (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:12.388 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T02:19:12.414 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:12.454 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ceph-mon (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:13.060 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-base. 2024-04-26T02:19:13.074 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T02:19:13.075 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:13.101 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:13.118 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-base (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:13.141 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ceph-osd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:13.665 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T02:19:13.691 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T02:19:13.731 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T02:19:14.209 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T02:19:14.236 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T02:19:14.276 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T02:19:14.403 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ceph. 2024-04-26T02:19:14.419 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../066-ceph_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:14.461 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:14.555 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T02:19:14.581 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T02:19:14.613 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T02:19:14.781 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T02:19:14.807 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:14.847 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ceph-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:14.917 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T02:19:14.943 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T02:19:14.983 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T02:19:15.159 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T02:19:15.185 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:15.225 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ceph-mds (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:15.294 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T02:19:15.321 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T02:19:15.361 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T02:19:15.612 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package cephadm. 2024-04-26T02:19:15.638 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../069-cephadm_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:15.647 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T02:19:15.674 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T02:19:15.678 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:15.706 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T02:19:15.997 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T02:19:16.009 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-portend. 2024-04-26T02:19:16.024 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T02:19:16.036 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T02:19:16.072 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T02:19:16.075 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T02:19:16.371 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T02:19:16.398 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T02:19:16.399 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T02:19:16.415 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T02:19:16.437 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T02:19:16.458 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T02:19:16.699 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T02:19:16.714 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T02:19:16.749 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T02:19:17.086 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T02:19:17.113 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T02:19:17.153 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T02:19:17.423 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T02:19:17.438 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T02:19:17.463 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-babel. 2024-04-26T02:19:17.473 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T02:19:17.478 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T02:19:17.513 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T02:19:17.701 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T02:19:17.728 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T02:19:17.767 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T02:19:17.799 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T02:19:17.826 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T02:19:17.882 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T02:19:18.079 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-mako. 2024-04-26T02:19:18.106 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T02:19:18.146 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T02:19:18.185 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T02:19:18.212 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:19:18.252 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:18.474 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T02:19:18.501 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T02:19:18.541 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T02:19:18.547 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T02:19:18.562 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T02:19:18.597 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T02:19:18.811 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T02:19:18.838 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T02:19:18.869 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T02:19:18.916 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-routes. 2024-04-26T02:19:18.943 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T02:19:18.982 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T02:19:19.155 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-webob. 2024-04-26T02:19:19.182 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T02:19:19.222 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T02:19:19.319 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T02:19:19.346 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:19:19.394 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:19.559 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T02:19:19.585 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T02:19:19.629 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T02:19:19.937 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T02:19:19.964 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T02:19:20.004 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T02:19:20.257 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-paste. 2024-04-26T02:19:20.284 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T02:19:20.323 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T02:19:20.662 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T02:19:20.676 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T02:19:20.682 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T02:19:20.710 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T02:19:20.718 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T02:19:20.749 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T02:19:20.989 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T02:19:21.016 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T02:19:21.055 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T02:19:21.236 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T02:19:21.251 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T02:19:21.285 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T02:19:21.359 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T02:19:21.386 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T02:19:21.425 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T02:19:21.656 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T02:19:21.683 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T02:19:21.739 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T02:19:21.745 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T02:19:21.772 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T02:19:21.814 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T02:19:22.025 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T02:19:22.052 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T02:19:22.094 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T02:19:22.152 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T02:19:22.179 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T02:19:22.219 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T02:19:22.522 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-idna. 2024-04-26T02:19:22.549 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T02:19:22.597 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T02:19:22.850 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T02:19:22.878 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:19:22.917 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T02:19:22.944 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T02:19:22.984 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T02:19:23.002 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:23.304 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-requests. 2024-04-26T02:19:23.331 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T02:19:23.370 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T02:19:23.691 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T02:19:23.718 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T02:19:23.757 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T02:19:23.939 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T02:19:23.966 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T02:19:24.005 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T02:19:24.052 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T02:19:24.068 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:19:24.110 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:24.308 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T02:19:24.336 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T02:19:24.375 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T02:19:24.650 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T02:19:24.677 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:24.716 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:24.887 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T02:19:24.914 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T02:19:24.945 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T02:19:25.036 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T02:19:25.063 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T02:19:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T02:19:25.215 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T02:19:25.242 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T02:19:25.273 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T02:19:25.406 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T02:19:25.433 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T02:19:25.481 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T02:19:25.610 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T02:19:25.637 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T02:19:25.676 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T02:19:25.768 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T02:19:25.794 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:25.826 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:25.988 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T02:19:26.015 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T02:19:26.054 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T02:19:26.238 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T02:19:26.265 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:26.304 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mon (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:26.366 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T02:19:26.393 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T02:19:26.432 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T02:19:26.735 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T02:19:26.762 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T02:19:26.802 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T02:19:26.933 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T02:19:26.960 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:27.000 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-osd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:27.441 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T02:19:27.457 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:19:27.491 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:27.802 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T02:19:27.829 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T02:19:27.868 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T02:19:28.045 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph. 2024-04-26T02:19:28.061 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../066-ceph_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:28.103 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:28.213 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T02:19:28.241 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T02:19:28.296 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T02:19:28.423 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T02:19:28.450 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:28.490 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:28.557 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package jq. 2024-04-26T02:19:28.585 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T02:19:28.624 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T02:19:28.851 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package socat. 2024-04-26T02:19:28.860 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T02:19:28.867 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T02:19:28.886 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:28.901 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T02:19:28.926 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mds (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:29.204 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T02:19:29.220 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T02:19:29.254 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T02:19:29.324 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package cephadm. 2024-04-26T02:19:29.351 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../069-cephadm_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:29.388 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:29.706 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ceph-test. 2024-04-26T02:19:29.733 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:29.783 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ceph-test (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:29.866 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T02:19:29.893 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T02:19:29.933 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T02:19:30.244 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T02:19:30.287 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T02:19:30.311 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T02:19:31.350 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-babel. 2024-04-26T02:19:31.377 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T02:19:31.416 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T02:19:31.736 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T02:19:31.764 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T02:19:31.803 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T02:19:31.980 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T02:19:32.007 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:19:32.046 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ceph-volume (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:32.139 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T02:19:32.155 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:19:32.206 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:32.358 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T02:19:32.374 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:32.416 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libcephfs-dev (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:32.544 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T02:19:32.571 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T02:19:32.610 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T02:19:32.702 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T02:19:32.718 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T02:19:32.760 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T02:19:32.946 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-routes. 2024-04-26T02:19:32.973 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T02:19:33.013 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T02:19:33.141 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T02:19:33.168 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T02:19:33.248 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T02:19:33.333 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T02:19:33.354 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:19:33.391 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:33.619 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T02:19:33.647 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T02:19:33.686 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T02:19:34.124 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package pkg-config. 2024-04-26T02:19:34.152 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T02:19:34.191 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T02:19:34.427 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T02:19:34.442 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T02:19:34.485 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T02:19:34.639 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T02:19:34.666 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T02:19:34.888 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T02:19:34.905 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T02:19:34.915 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T02:19:34.955 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T02:19:35.254 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T02:19:35.281 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T02:19:35.324 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T02:19:35.409 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T02:19:35.436 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T02:19:35.476 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T02:19:35.610 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T02:19:35.626 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T02:19:35.661 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T02:19:35.855 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T02:19:35.882 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T02:19:35.921 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T02:19:35.947 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T02:19:35.974 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T02:19:36.013 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T02:19:36.241 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T02:19:36.268 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T02:19:36.307 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T02:19:36.325 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-py. 2024-04-26T02:19:36.352 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T02:19:36.399 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T02:19:36.711 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T02:19:36.739 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T02:19:36.778 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T02:19:37.009 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T02:19:37.037 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:19:37.161 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:37.248 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T02:19:37.276 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T02:19:37.315 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T02:19:37.634 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-toml. 2024-04-26T02:19:37.661 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T02:19:37.701 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T02:19:37.970 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T02:19:37.997 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T02:19:38.028 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T02:19:38.190 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T02:19:38.217 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T02:19:38.257 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T02:19:38.349 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T02:19:38.376 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T02:19:38.415 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T02:19:38.560 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T02:19:38.587 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T02:19:38.626 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T02:19:38.692 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T02:19:38.708 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T02:19:38.750 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T02:19:38.972 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T02:19:38.999 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T02:19:39.038 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T02:19:39.350 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T02:19:39.378 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T02:19:39.417 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T02:19:39.455 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package radosgw. 2024-04-26T02:19:39.482 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../116-radosgw_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:39.529 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking radosgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:39.771 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T02:19:39.798 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T02:19:39.846 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T02:19:40.233 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T02:19:40.261 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T02:19:40.311 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T02:19:40.591 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T02:19:40.606 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:40.641 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking rbd-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:40.653 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T02:19:40.680 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T02:19:40.720 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T02:19:40.936 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T02:19:41.032 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T02:19:41.059 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T02:19:41.098 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T02:19:41.204 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T02:19:41.338 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T02:19:41.576 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T02:19:41.838 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T02:19:41.866 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:19:41.895 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T02:19:41.904 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:42.241 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T02:19:42.268 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T02:19:42.308 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T02:19:42.366 INFO:teuthology.orchestra.run.smithi087.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T02:19:42.432 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:42.603 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T02:19:42.630 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T02:19:42.669 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T02:19:42.898 INFO:teuthology.orchestra.run.smithi087.stdout:Adding system user cephadm....done 2024-04-26T02:19:42.930 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package jq. 2024-04-26T02:19:42.946 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T02:19:42.989 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T02:19:43.029 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T02:19:43.225 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package socat. 2024-04-26T02:19:43.241 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T02:19:43.283 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T02:19:43.314 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T02:19:43.557 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T02:19:43.570 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T02:19:43.586 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T02:19:43.620 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T02:19:43.683 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T02:19:43.934 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T02:19:43.966 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-test. 2024-04-26T02:19:43.983 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:44.025 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-test (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:44.185 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T02:19:44.311 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T02:19:44.613 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T02:19:44.965 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T02:19:45.208 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T02:19:45.509 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T02:19:45.769 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T02:19:46.020 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T02:19:46.280 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T02:19:46.339 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T02:19:46.366 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T02:19:46.405 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-volume (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:46.522 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-ceph-argparse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:46.667 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T02:19:46.683 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:46.742 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libcephfs-dev (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:46.850 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T02:19:46.966 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T02:19:47.054 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T02:19:47.081 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T02:19:47.083 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T02:19:47.120 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T02:19:47.210 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T02:19:47.459 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T02:19:47.477 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T02:19:47.486 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T02:19:47.525 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T02:19:47.830 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T02:19:47.857 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T02:19:47.866 INFO:teuthology.orchestra.run.smithi087.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T02:19:47.888 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T02:19:47.902 INFO:teuthology.orchestra.run.smithi087.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T02:19:47.964 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T02:19:48.065 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T02:19:48.302 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package pkg-config. 2024-04-26T02:19:48.330 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T02:19:48.369 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T02:19:48.433 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T02:19:48.534 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T02:19:48.622 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T02:19:48.649 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T02:19:48.688 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T02:19:48.818 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T02:19:49.095 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T02:19:49.126 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T02:19:49.153 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T02:19:49.192 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T02:19:49.413 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T02:19:49.495 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T02:19:49.522 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T02:19:49.530 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T02:19:49.562 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T02:19:49.865 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T02:19:49.893 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T02:19:49.932 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T02:19:50.165 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T02:19:50.219 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T02:19:50.246 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T02:19:50.285 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T02:19:50.433 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T02:19:50.559 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T02:19:50.639 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-py. 2024-04-26T02:19:50.666 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T02:19:50.685 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T02:19:50.705 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T02:19:50.978 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T02:19:51.017 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T02:19:51.045 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T02:19:51.084 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T02:19:51.357 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T02:19:51.513 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T02:19:51.541 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T02:19:51.580 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T02:19:51.591 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T02:19:51.891 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-toml. 2024-04-26T02:19:51.919 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T02:19:51.942 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T02:19:51.957 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T02:19:52.210 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T02:19:52.261 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T02:19:52.288 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T02:19:52.327 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T02:19:52.587 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T02:19:52.681 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T02:19:52.709 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T02:19:52.748 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T02:19:52.837 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T02:19:52.963 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:53.026 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T02:19:53.054 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T02:19:53.092 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T02:19:53.240 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T02:19:53.549 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T02:19:53.889 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package radosgw. 2024-04-26T02:19:53.916 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../116-radosgw_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:53.955 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking radosgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:54.738 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T02:19:54.808 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T02:19:54.825 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T02:19:54.866 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking rbd-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:54.973 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:19:55.082 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T02:19:55.146 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T02:19:55.325 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T02:19:55.405 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T02:19:55.434 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T02:19:55.532 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T02:19:55.535 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T02:19:55.792 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T02:19:55.994 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T02:19:56.143 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T02:19:56.290 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T02:19:56.543 INFO:teuthology.orchestra.run.smithi122.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T02:19:56.572 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:19:56.609 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:19:56.674 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T02:19:56.966 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T02:19:57.151 INFO:teuthology.orchestra.run.smithi122.stdout:Adding system user cephadm....done 2024-04-26T02:19:57.217 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T02:19:57.307 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T02:19:57.476 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T02:19:57.584 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T02:19:57.727 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T02:19:57.852 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T02:19:57.953 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T02:19:57.978 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T02:19:58.188 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T02:19:58.237 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T02:19:58.363 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T02:19:58.439 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T02:19:58.565 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T02:19:58.640 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T02:19:58.766 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T02:19:58.876 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T02:19:59.066 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T02:19:59.220 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T02:19:59.384 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T02:19:59.488 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T02:19:59.635 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T02:19:59.756 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T02:19:59.761 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T02:19:59.887 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T02:19:59.977 INFO:teuthology.orchestra.run.smithi087.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T02:19:59.999 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T02:20:00.205 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T02:20:00.259 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T02:20:00.315 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T02:20:00.502 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T02:20:00.700 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T02:20:00.942 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T02:20:01.068 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T02:20:01.078 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-ceph-argparse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:01.327 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:20:01.437 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T02:20:01.453 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T02:20:01.555 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T02:20:01.681 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T02:20:01.771 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T02:20:01.807 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T02:20:02.064 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T02:20:02.075 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T02:20:02.332 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T02:20:02.433 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T02:20:02.447 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T02:20:02.476 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T02:20:02.539 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T02:20:02.665 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T02:20:02.700 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T02:20:02.968 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T02:20:03.041 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T02:20:03.167 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T02:20:03.330 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T02:20:03.418 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T02:20:03.678 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T02:20:03.723 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T02:20:03.979 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T02:20:03.999 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T02:20:04.088 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T02:20:04.318 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T02:20:04.444 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up librados2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:04.570 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:04.696 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T02:20:04.732 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T02:20:04.991 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T02:20:05.032 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libcephfs2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:05.117 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T02:20:05.150 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libradosstriper1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:05.243 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T02:20:05.259 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T02:20:05.519 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up librbd1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:05.527 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T02:20:05.645 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ceph-mgr-modules-core (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:05.771 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ceph-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:05.916 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T02:20:05.978 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T02:20:05.978 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T02:20:06.155 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T02:20:06.472 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libcephfs-dev (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:06.515 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T02:20:06.573 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-rados (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:06.682 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up librgw2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:06.783 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T02:20:06.816 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T02:20:07.159 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T02:20:07.393 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T02:20:07.503 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:07.787 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T02:20:08.020 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-rbd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:08.088 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T02:20:08.146 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up rbd-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:08.272 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-rgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:08.398 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up python3-cephfs (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:08.524 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:09.008 INFO:teuthology.orchestra.run.smithi087.stdout:Adding group ceph....done 2024-04-26T02:20:09.244 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T02:20:09.384 INFO:teuthology.orchestra.run.smithi087.stdout:Adding system user ceph....done 2024-04-26T02:20:09.520 INFO:teuthology.orchestra.run.smithi087.stdout:Setting system user ceph properties....done 2024-04-26T02:20:09.521 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:20:09.532 INFO:teuthology.orchestra.run.smithi087.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T02:20:09.647 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T02:20:09.667 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T02:20:09.914 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T02:20:09.971 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T02:20:10.049 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T02:20:10.175 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T02:20:10.468 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T02:20:10.475 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ceph-test (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:10.601 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up radosgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:10.735 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T02:20:10.987 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:20:11.086 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T02:20:11.086 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T02:20:11.113 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T02:20:11.439 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T02:20:11.577 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ceph-base (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:11.698 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T02:20:11.944 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T02:20:11.966 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T02:20:12.234 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T02:20:12.454 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ceph-mds (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:12.476 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T02:20:12.682 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T02:20:12.682 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T02:20:12.736 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T02:20:12.870 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T02:20:13.155 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T02:20:13.197 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ceph-mgr (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:13.281 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T02:20:13.420 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T02:20:13.420 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T02:20:13.574 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T02:20:13.875 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T02:20:13.907 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ceph-osd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:14.126 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T02:20:14.171 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T02:20:14.171 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T02:20:14.227 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T02:20:14.328 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T02:20:14.425 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T02:20:14.646 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T02:20:14.692 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:14.772 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T02:20:14.818 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:14.969 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ceph-mon (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:15.165 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T02:20:15.186 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T02:20:15.186 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T02:20:15.425 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T02:20:15.550 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T02:20:15.662 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ceph-mgr-cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:15.780 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:15.801 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T02:20:15.880 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ceph-mgr-dashboard (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:15.911 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T02:20:16.023 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ceph-volume (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:16.237 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T02:20:16.377 INFO:teuthology.orchestra.run.smithi087.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T02:20:16.486 INFO:teuthology.orchestra.run.smithi087.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T02:20:16.638 INFO:teuthology.orchestra.run.smithi087.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T02:20:16.663 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T02:20:16.931 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T02:20:17.057 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T02:20:17.333 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T02:20:17.609 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T02:20:17.680 INFO:teuthology.orchestra.run.smithi087.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T02:20:17.946 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T02:20:18.294 INFO:teuthology.orchestra.run.smithi087.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T02:20:18.297 DEBUG:teuthology.orchestra.run.smithi087:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install sqlite3 2024-04-26T02:20:18.314 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T02:20:18.350 INFO:teuthology.orchestra.run.smithi087.stdout:Reading package lists... 2024-04-26T02:20:18.551 INFO:teuthology.orchestra.run.smithi087.stdout:Building dependency tree... 2024-04-26T02:20:18.552 INFO:teuthology.orchestra.run.smithi087.stdout:Reading state information... 2024-04-26T02:20:18.599 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T02:20:18.743 INFO:teuthology.orchestra.run.smithi087.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T02:20:18.743 INFO:teuthology.orchestra.run.smithi087.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T02:20:18.744 INFO:teuthology.orchestra.run.smithi087.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T02:20:18.744 INFO:teuthology.orchestra.run.smithi087.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T02:20:18.744 INFO:teuthology.orchestra.run.smithi087.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T02:20:18.744 INFO:teuthology.orchestra.run.smithi087.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T02:20:18.744 INFO:teuthology.orchestra.run.smithi087.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T02:20:18.745 INFO:teuthology.orchestra.run.smithi087.stdout:The following additional packages will be installed: 2024-04-26T02:20:18.746 INFO:teuthology.orchestra.run.smithi087.stdout: libsqlite3-0 2024-04-26T02:20:18.747 INFO:teuthology.orchestra.run.smithi087.stdout:Suggested packages: 2024-04-26T02:20:18.747 INFO:teuthology.orchestra.run.smithi087.stdout: sqlite3-doc 2024-04-26T02:20:18.777 INFO:teuthology.orchestra.run.smithi087.stdout:The following NEW packages will be installed: 2024-04-26T02:20:18.777 INFO:teuthology.orchestra.run.smithi087.stdout: sqlite3 2024-04-26T02:20:18.778 INFO:teuthology.orchestra.run.smithi087.stdout:The following packages will be upgraded: 2024-04-26T02:20:18.779 INFO:teuthology.orchestra.run.smithi087.stdout: libsqlite3-0 2024-04-26T02:20:18.959 INFO:teuthology.orchestra.run.smithi087.stdout:1 upgraded, 1 newly installed, 0 to remove and 301 not upgraded. 2024-04-26T02:20:18.959 INFO:teuthology.orchestra.run.smithi087.stdout:Need to get 1,410 kB of archives. 2024-04-26T02:20:18.959 INFO:teuthology.orchestra.run.smithi087.stdout:After this operation, 1,873 kB of additional disk space will be used. 2024-04-26T02:20:18.959 INFO:teuthology.orchestra.run.smithi087.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libsqlite3-0 amd64 3.37.2-2ubuntu0.3 [641 kB] 2024-04-26T02:20:18.993 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T02:20:19.102 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librados2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:19.229 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:19.355 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T02:20:19.532 INFO:teuthology.orchestra.run.smithi087.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 sqlite3 amd64 3.37.2-2ubuntu0.3 [768 kB] 2024-04-26T02:20:19.691 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libcephfs2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:19.817 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libradosstriper1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:19.943 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T02:20:19.991 INFO:teuthology.orchestra.run.smithi087.stdout:Fetched 1,410 kB in 1s (1,679 kB/s) 2024-04-26T02:20:20.219 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librbd1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:20.255 INFO:teuthology.orchestra.run.smithi087.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173891 files and directories currently installed.) 2024-04-26T02:20:20.261 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../libsqlite3-0_3.37.2-2ubuntu0.3_amd64.deb ... 2024-04-26T02:20:20.337 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-modules-core (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:20.434 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libsqlite3-0:amd64 (3.37.2-2ubuntu0.3) over (3.37.2-2ubuntu0.1) ... 2024-04-26T02:20:20.463 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:20.676 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T02:20:20.676 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T02:20:20.762 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package sqlite3. 2024-04-26T02:20:20.778 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../sqlite3_3.37.2-2ubuntu0.3_amd64.deb ... 2024-04-26T02:20:20.812 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking sqlite3 (3.37.2-2ubuntu0.3) ... 2024-04-26T02:20:21.123 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libcephfs-dev (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:21.155 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libsqlite3-0:amd64 (3.37.2-2ubuntu0.3) ... 2024-04-26T02:20:21.249 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rados (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:21.282 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up sqlite3 (3.37.2-2ubuntu0.3) ... 2024-04-26T02:20:21.375 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librgw2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:21.501 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T02:20:21.502 INFO:teuthology.orchestra.run.smithi087.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T02:20:22.527 INFO:teuthology.orchestra.run.smithi087.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T02:20:22.705 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rbd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:22.831 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up rbd-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:22.957 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:22.972 INFO:teuthology.orchestra.run.smithi087.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T02:20:22.978 DEBUG:teuthology.parallel:result is None 2024-04-26T02:20:23.083 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cephfs (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:23.225 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:23.710 INFO:teuthology.orchestra.run.smithi122.stdout:Adding group ceph....done 2024-04-26T02:20:24.119 INFO:teuthology.orchestra.run.smithi122.stdout:Adding system user ceph....done 2024-04-26T02:20:24.279 INFO:teuthology.orchestra.run.smithi122.stdout:Setting system user ceph properties....done 2024-04-26T02:20:24.295 INFO:teuthology.orchestra.run.smithi122.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T02:20:24.431 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T02:20:24.796 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T02:20:25.276 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-test (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:25.385 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up radosgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:25.836 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T02:20:25.836 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T02:20:26.296 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-base (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:26.637 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T02:20:27.098 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mds (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:27.325 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T02:20:27.325 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T02:20:27.783 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:28.016 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T02:20:28.017 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T02:20:28.509 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-osd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:28.788 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T02:20:28.788 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T02:20:29.387 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:29.546 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:29.697 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mon (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:29.909 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T02:20:29.909 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T02:20:30.390 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:30.541 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:30.667 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-dashboard (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:30.818 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-volume (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T02:20:31.154 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T02:20:31.255 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T02:20:31.408 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T02:20:32.550 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T02:20:33.205 INFO:teuthology.orchestra.run.smithi122.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T02:20:33.208 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install sqlite3 2024-04-26T02:20:33.262 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2024-04-26T02:20:33.472 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2024-04-26T02:20:33.472 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2024-04-26T02:20:33.662 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T02:20:33.662 INFO:teuthology.orchestra.run.smithi122.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T02:20:33.662 INFO:teuthology.orchestra.run.smithi122.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T02:20:33.662 INFO:teuthology.orchestra.run.smithi122.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T02:20:33.662 INFO:teuthology.orchestra.run.smithi122.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T02:20:33.663 INFO:teuthology.orchestra.run.smithi122.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T02:20:33.663 INFO:teuthology.orchestra.run.smithi122.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T02:20:33.663 INFO:teuthology.orchestra.run.smithi122.stdout:The following additional packages will be installed: 2024-04-26T02:20:33.664 INFO:teuthology.orchestra.run.smithi122.stdout: libsqlite3-0 2024-04-26T02:20:33.665 INFO:teuthology.orchestra.run.smithi122.stdout:Suggested packages: 2024-04-26T02:20:33.665 INFO:teuthology.orchestra.run.smithi122.stdout: sqlite3-doc 2024-04-26T02:20:33.693 INFO:teuthology.orchestra.run.smithi122.stdout:The following NEW packages will be installed: 2024-04-26T02:20:33.694 INFO:teuthology.orchestra.run.smithi122.stdout: sqlite3 2024-04-26T02:20:33.694 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages will be upgraded: 2024-04-26T02:20:33.695 INFO:teuthology.orchestra.run.smithi122.stdout: libsqlite3-0 2024-04-26T02:20:33.752 INFO:teuthology.orchestra.run.smithi122.stdout:1 upgraded, 1 newly installed, 0 to remove and 301 not upgraded. 2024-04-26T02:20:33.752 INFO:teuthology.orchestra.run.smithi122.stdout:Need to get 1,410 kB of archives. 2024-04-26T02:20:33.752 INFO:teuthology.orchestra.run.smithi122.stdout:After this operation, 1,873 kB of additional disk space will be used. 2024-04-26T02:20:33.752 INFO:teuthology.orchestra.run.smithi122.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libsqlite3-0 amd64 3.37.2-2ubuntu0.3 [641 kB] 2024-04-26T02:20:33.899 INFO:teuthology.orchestra.run.smithi122.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 sqlite3 amd64 3.37.2-2ubuntu0.3 [768 kB] 2024-04-26T02:20:34.280 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 1,410 kB in 0s (6,371 kB/s) 2024-04-26T02:20:34.544 INFO:teuthology.orchestra.run.smithi122.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173891 files and directories currently installed.) 2024-04-26T02:20:34.549 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../libsqlite3-0_3.37.2-2ubuntu0.3_amd64.deb ... 2024-04-26T02:20:34.722 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libsqlite3-0:amd64 (3.37.2-2ubuntu0.3) over (3.37.2-2ubuntu0.1) ... 2024-04-26T02:20:35.034 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package sqlite3. 2024-04-26T02:20:35.050 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../sqlite3_3.37.2-2ubuntu0.3_amd64.deb ... 2024-04-26T02:20:35.084 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking sqlite3 (3.37.2-2ubuntu0.3) ... 2024-04-26T02:20:35.436 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libsqlite3-0:amd64 (3.37.2-2ubuntu0.3) ... 2024-04-26T02:20:35.563 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up sqlite3 (3.37.2-2ubuntu0.3) ... 2024-04-26T02:20:35.706 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T02:20:36.748 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T02:20:37.328 INFO:teuthology.orchestra.run.smithi122.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T02:20:37.333 DEBUG:teuthology.parallel:result is None 2024-04-26T02:20:37.334 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T02:20:37.481 DEBUG:teuthology.orchestra.run.smithi087:> dpkg-query -W -f '${Version}' ceph 2024-04-26T02:20:37.501 INFO:teuthology.orchestra.run.smithi087.stdout:19.0.0-2457-gddeaa917-1jammy 2024-04-26T02:20:37.501 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2457-gddeaa917-1jammy 2024-04-26T02:20:37.501 INFO:teuthology.task.install:The correct ceph version 19.0.0-2457-gddeaa917-1jammy is installed. 2024-04-26T02:20:37.503 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T02:20:37.642 DEBUG:teuthology.orchestra.run.smithi122:> dpkg-query -W -f '${Version}' ceph 2024-04-26T02:20:37.661 INFO:teuthology.orchestra.run.smithi122.stdout:19.0.0-2457-gddeaa917-1jammy 2024-04-26T02:20:37.661 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2457-gddeaa917-1jammy 2024-04-26T02:20:37.661 INFO:teuthology.task.install:The correct ceph version 19.0.0-2457-gddeaa917-1jammy is installed. 2024-04-26T02:20:37.663 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T02:20:37.663 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:20:37.663 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T02:20:37.677 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:20:37.677 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T02:20:37.715 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T02:20:37.716 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:20:37.716 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T02:20:37.730 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T02:20:37.786 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:20:37.786 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T02:20:37.800 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T02:20:37.854 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T02:20:37.854 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:20:37.854 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T02:20:37.867 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T02:20:37.926 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:20:37.927 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T02:20:37.938 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T02:20:37.993 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T02:20:37.993 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:20:37.994 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T02:20:38.007 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T02:20:38.062 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:20:38.062 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T02:20:38.075 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T02:20:38.130 INFO:teuthology.run_tasks:Running task ceph... 2024-04-26T02:20:38.217 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-26T02:20:38.217 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 777 /var/log/ceph 2024-04-26T02:20:38.219 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /var/log/ceph 2024-04-26T02:20:38.232 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-26T02:20:38.232 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T02:20:38.276 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T02:20:38.288 INFO:tasks.ceph:Creating extra log directories... 2024-04-26T02:20:38.288 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T02:20:38.332 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T02:20:38.354 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-26T02:20:38.354 INFO:tasks.ceph:config {'conf': {'client': {'debug cephsqlite': 20, 'debug client': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 1500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-04-26T02:20:38.355 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674282', 'branch': 'wip-yuri2-testing-2024-04-24-0914-squid', 'description': 'rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7674282', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri2-testing-2024-04-24-0914-squid'}, 'ceph': {'conf': {'client': {'debug cephsqlite': 20, 'debug client': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 1500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled'], 'mon_bind_msgr2': False, 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri2-testing-2024-04-24-0914-squid', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 1848, 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri2-testing-2024-04-24-0914-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448', 'targets': {'smithi087.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMyKkTW4yPxjq/fNiQrlOxp4P5aAVaV2x3BU5ji/rAibHmTisV0URzcbl8/WtBPYn+9IfByO+47YWaythDc/tMo=', 'smithi122.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPPjC4Gp4IcpOCh1ISncnKzkhVM1UAGkyhgg83Xam6I3XhAPDIWfBR5lFiXozObI3bKw86iNzsYG9BStDKdUdsc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_system_packages': {'deb': ['sqlite3'], 'rpm': ['sqlite-devel']}, 'flavor': 'default', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'}}, {'ceph': None}, {'exec': {'client.0': ['ceph osd pool create cephsqlite', "ceph auth get-or-create client.libcephsqlite mon 'profile simple-rados-client-with-blocklist' osd 'allow rwx pool=cephsqlite' >> /etc/ceph/ceph.keyring"]}}, {'exec': {'client.0': ['ceph_test_libcephsqlite --id libcephsqlite --no-log-to-stderr']}}, {'workunit': {'clients': {'client.0': ['rados/test_libcephsqlite.sh cephsqlite']}, 'env': {'CEPH_ARGS': '--id libcephsqlite --no-log-to-stderr'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-26_01:23:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-26T02:20:38.355 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T02:20:38.388 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T02:20:38.393 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T02:20:38.435 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T02:20:38.448 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:20:38.448 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T02:20:38.495 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T02:20:38.495 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_1 2024-04-26T02:20:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T02:20:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T02:20:38.543 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 870 Links: 1 2024-04-26T02:20:38.543 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T02:20:38.543 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-04-26 02:15:03.498301805 +0000 2024-04-26T02:20:38.543 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-04-26 02:15:03.210304378 +0000 2024-04-26T02:20:38.543 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-04-26 02:15:03.210304378 +0000 2024-04-26T02:20:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2024-04-26T02:20:38.544 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T02:20:38.597 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-04-26T02:20:38.597 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-04-26T02:20:38.597 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000335703 s, 1.5 MB/s 2024-04-26T02:20:38.598 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T02:20:38.648 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_2 2024-04-26T02:20:38.695 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T02:20:38.695 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T02:20:38.695 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 894 Links: 1 2024-04-26T02:20:38.695 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T02:20:38.695 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-04-26 02:15:03.974297552 +0000 2024-04-26T02:20:38.695 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-04-26 02:15:03.726299768 +0000 2024-04-26T02:20:38.695 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-04-26 02:15:03.726299768 +0000 2024-04-26T02:20:38.695 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2024-04-26T02:20:38.695 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T02:20:38.749 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-04-26T02:20:38.749 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-04-26T02:20:38.749 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000453748 s, 1.1 MB/s 2024-04-26T02:20:38.750 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T02:20:38.800 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_3 2024-04-26T02:20:38.847 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T02:20:38.847 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T02:20:38.847 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-04-26T02:20:38.847 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T02:20:38.847 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-04-26 02:15:04.526292622 +0000 2024-04-26T02:20:38.847 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-04-26 02:15:04.214295410 +0000 2024-04-26T02:20:38.847 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-04-26 02:15:04.214295410 +0000 2024-04-26T02:20:38.847 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2024-04-26T02:20:38.847 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T02:20:38.901 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-04-26T02:20:38.901 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-04-26T02:20:38.901 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000346645 s, 1.5 MB/s 2024-04-26T02:20:38.902 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T02:20:38.953 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_4 2024-04-26T02:20:38.998 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T02:20:38.998 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T02:20:38.998 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 922 Links: 1 2024-04-26T02:20:38.999 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T02:20:38.999 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-04-26 02:15:05.010288299 +0000 2024-04-26T02:20:38.999 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-04-26 02:15:04.746290657 +0000 2024-04-26T02:20:38.999 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-04-26 02:15:04.746290657 +0000 2024-04-26T02:20:38.999 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2024-04-26T02:20:38.999 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T02:20:39.053 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-04-26T02:20:39.053 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-04-26T02:20:39.053 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000328666 s, 1.6 MB/s 2024-04-26T02:20:39.054 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T02:20:39.104 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T02:20:39.104 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:20:39.104 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T02:20:39.110 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T02:20:39.110 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2024-04-26T02:20:39.159 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T02:20:39.159 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T02:20:39.159 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 871 Links: 1 2024-04-26T02:20:39.159 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T02:20:39.159 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-26 02:15:11.185261448 +0000 2024-04-26T02:20:39.160 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-26 02:15:10.933263698 +0000 2024-04-26T02:20:39.160 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-26 02:15:10.933263698 +0000 2024-04-26T02:20:39.160 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2024-04-26T02:20:39.160 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T02:20:39.213 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-26T02:20:39.214 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-26T02:20:39.214 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.00041534 s, 1.2 MB/s 2024-04-26T02:20:39.215 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T02:20:39.264 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2024-04-26T02:20:39.311 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T02:20:39.311 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T02:20:39.311 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 884 Links: 1 2024-04-26T02:20:39.311 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T02:20:39.311 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-26 02:15:11.665257159 +0000 2024-04-26T02:20:39.311 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-26 02:15:11.425259304 +0000 2024-04-26T02:20:39.311 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-26 02:15:11.425259304 +0000 2024-04-26T02:20:39.311 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2024-04-26T02:20:39.311 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T02:20:39.365 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-26T02:20:39.365 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-26T02:20:39.365 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000425694 s, 1.2 MB/s 2024-04-26T02:20:39.367 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T02:20:39.416 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2024-04-26T02:20:39.463 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T02:20:39.463 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T02:20:39.463 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-04-26T02:20:39.463 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T02:20:39.463 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-26 02:15:12.173252622 +0000 2024-04-26T02:20:39.463 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-26 02:15:11.881255231 +0000 2024-04-26T02:20:39.463 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-26 02:15:11.881255231 +0000 2024-04-26T02:20:39.463 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2024-04-26T02:20:39.463 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T02:20:39.517 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-26T02:20:39.517 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-26T02:20:39.517 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000411003 s, 1.2 MB/s 2024-04-26T02:20:39.519 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T02:20:39.568 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2024-04-26T02:20:39.615 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T02:20:39.615 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T02:20:39.615 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 921 Links: 1 2024-04-26T02:20:39.615 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T02:20:39.616 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-26 02:15:12.417250444 +0000 2024-04-26T02:20:39.616 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-26 02:15:12.413250480 +0000 2024-04-26T02:20:39.616 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-26 02:15:12.413250480 +0000 2024-04-26T02:20:39.616 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2024-04-26T02:20:39.616 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T02:20:39.669 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-26T02:20:39.670 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-26T02:20:39.670 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.00040119 s, 1.3 MB/s 2024-04-26T02:20:39.671 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T02:20:39.720 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-26T02:20:39.720 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-26T02:20:39.720 INFO:tasks.ceph:Generating config... 2024-04-26T02:20:39.722 INFO:tasks.ceph:[client] debug cephsqlite = 20 2024-04-26T02:20:39.722 INFO:tasks.ceph:[client] debug client = 20 2024-04-26T02:20:39.722 INFO:tasks.ceph:[client] debug ms = 1 2024-04-26T02:20:39.722 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-26T02:20:39.722 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-04-26T02:20:39.722 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-04-26T02:20:39.722 INFO:tasks.ceph:[global] ms inject socket failures = 1500 2024-04-26T02:20:39.722 INFO:tasks.ceph:[global] ms type = async 2024-04-26T02:20:39.722 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-26T02:20:39.723 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-26T02:20:39.723 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-26T02:20:39.723 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-26T02:20:39.723 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-26T02:20:39.723 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-26T02:20:39.723 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-26T02:20:39.723 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-26T02:20:39.723 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-26T02:20:39.723 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-26T02:20:39.723 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-26T02:20:39.723 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-26T02:20:39.723 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-04-26T02:20:39.723 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-26T02:20:39.724 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-26T02:20:39.724 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-26T02:20:39.724 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-26T02:20:39.724 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-26T02:20:39.724 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-26T02:20:39.724 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-26T02:20:39.724 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-26T02:20:39.724 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-26T02:20:39.724 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-26T02:20:39.724 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-26T02:20:39.724 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-26T02:20:39.724 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-26T02:20:39.724 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-26T02:20:39.725 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-26T02:20:39.725 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-26T02:20:39.725 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-26T02:20:39.725 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-26T02:20:39.725 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-26T02:20:39.725 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-26T02:20:39.725 INFO:tasks.ceph:Setting up mon.a... 2024-04-26T02:20:39.726 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-26T02:20:39.758 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/ceph.keyring 2024-04-26T02:20:39.779 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-26T02:20:39.858 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T02:20:39.914 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.87:6789]'), ('mon.c', '[v1:172.21.15.87:6790]'), ('mon.b', '[v1:172.21.15.122:6789]')] 2024-04-26T02:20:39.915 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.87:6789],[v1:172.21.15.87:6790],[v1:172.21.15.122:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 1500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug cephsqlite': 20, 'debug client': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-26T02:20:39.915 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:20:39.916 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T02:20:39.963 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.87:6789]' --addv c '[v1:172.21.15.87:6790]' --addv b '[v1:172.21.15.122:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:20:40.048 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:20:40.048 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: generated fsid 6232cbab-48ed-4660-9de8-286f4c0e9a32 2024-04-26T02:20:40.049 INFO:teuthology.orchestra.run.smithi087.stdout:setting min_mon_release = quincy 2024-04-26T02:20:40.049 INFO:teuthology.orchestra.run.smithi087.stdout:epoch 0 2024-04-26T02:20:40.049 INFO:teuthology.orchestra.run.smithi087.stdout:fsid 6232cbab-48ed-4660-9de8-286f4c0e9a32 2024-04-26T02:20:40.049 INFO:teuthology.orchestra.run.smithi087.stdout:last_changed 2024-04-26T02:20:40.052216+0000 2024-04-26T02:20:40.049 INFO:teuthology.orchestra.run.smithi087.stdout:created 2024-04-26T02:20:40.052216+0000 2024-04-26T02:20:40.049 INFO:teuthology.orchestra.run.smithi087.stdout:min_mon_release 17 (quincy) 2024-04-26T02:20:40.049 INFO:teuthology.orchestra.run.smithi087.stdout:election_strategy: 1 2024-04-26T02:20:40.049 INFO:teuthology.orchestra.run.smithi087.stdout:0: v1:172.21.15.87:6789/0 mon.a 2024-04-26T02:20:40.049 INFO:teuthology.orchestra.run.smithi087.stdout:1: v1:172.21.15.122:6789/0 mon.b 2024-04-26T02:20:40.049 INFO:teuthology.orchestra.run.smithi087.stdout:2: v1:172.21.15.87:6790/0 mon.c 2024-04-26T02:20:40.049 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-26T02:20:40.051 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T02:20:40.099 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6232cbab-48ed-4660-9de8-286f4c0e9a32... 2024-04-26T02:20:40.099 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T02:20:40.143 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T02:20:40.172 INFO:teuthology.orchestra.run.smithi087.stdout:[global] 2024-04-26T02:20:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: chdir = "" 2024-04-26T02:20:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T02:20:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: auth supported = cephx 2024-04-26T02:20:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: filestore xattr use omap = true 2024-04-26T02:20:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: mon clock drift allowed = 1.000 2024-04-26T02:20:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: osd crush chooseleaf type = 0 2024-04-26T02:20:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: auth debug = true 2024-04-26T02:20:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on old message = true 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on bug = true 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: mon pg warn max object skew = 0 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default size = 2 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow primary affinity = true 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow pg remap = true 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on legacy crush tunables = false 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on crush straw calc version zero = false 2024-04-26T02:20:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on no sortbitwise = false 2024-04-26T02:20:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on osd down out interval zero = false 2024-04-26T02:20:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on too few osds = false 2024-04-26T02:20:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T02:20:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T02:20:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: mon_allow_pool_size_one = true 2024-04-26T02:20:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T02:20:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: osd default data pool replay window = 5 2024-04-26T02:20:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: mon allow pool delete = true 2024-04-26T02:20:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: mon cluster log file level = debug 2024-04-26T02:20:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: debug asserts on shutdown = true 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: mon health detail to clog = false 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: mon host = "[v1:172.21.15.87:6789],[v1:172.21.15.87:6790],[v1:172.21.15.122:6789]" 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: mon client directed command retry = 5 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: mon election default strategy = 1 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: ms bind msgr2 = False 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: ms inject socket failures = 1500 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: ms type = async 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: fsid = 6232cbab-48ed-4660-9de8-286f4c0e9a32 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout:[osd] 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: osd journal size = 100 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub load threshold = 5.0 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub max interval = 600 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: osd mclock profile = high_recovery_ops 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: osd recover clone overlap = true 2024-04-26T02:20:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: osd recovery max chunk = 1048576 2024-04-26T02:20:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug shutdown = true 2024-04-26T02:20:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug op order = true 2024-04-26T02:20:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify stray on activate = true 2024-04-26T02:20:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug trim objects = true 2024-04-26T02:20:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: osd open classes on start = true 2024-04-26T02:20:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug pg log writeout = true 2024-04-26T02:20:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: osd deep scrub update digest min age = 30 2024-04-26T02:20:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: osd map max advance = 10 2024-04-26T02:20:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: journal zero on create = true 2024-04-26T02:20:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: filestore ondisk finisher threads = 3 2024-04-26T02:20:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: filestore apply finisher threads = 3 2024-04-26T02:20:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: bdev debug aio = true 2024-04-26T02:20:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug misdirected ops = true 2024-04-26T02:20:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore block size = 96636764160 2024-04-26T02:20:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore compression algorithm = zstd 2024-04-26T02:20:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore compression mode = aggressive 2024-04-26T02:20:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore fsck on mount = True 2024-04-26T02:20:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore zero block detection = True 2024-04-26T02:20:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluefs = 20 2024-04-26T02:20:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluestore = 20 2024-04-26T02:20:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-04-26T02:20:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: debug osd = 20 2024-04-26T02:20:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: debug rocksdb = 10 2024-04-26T02:20:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T02:20:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd full ratio = 0.9 2024-04-26T02:20:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd nearfull ratio = 0.8 2024-04-26T02:20:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify cached snaps = True 2024-04-26T02:20:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify missing on start = True 2024-04-26T02:20:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: osd failsafe full ratio = 0.95 2024-04-26T02:20:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: osd mclock override recovery settings = True 2024-04-26T02:20:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: osd objectstore = bluestore 2024-04-26T02:20:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: osd op queue = debug_random 2024-04-26T02:20:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: osd op queue cut off = debug_random 2024-04-26T02:20:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: osd shutdown pgref assert = True 2024-04-26T02:20:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.178 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr] 2024-04-26T02:20:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: debug mgr = 20 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: mgr/telemetry/nag = false 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout:[mon] 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: debug paxos = 20 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: mon data avail warn = 5 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: mon mgr mkfs grace = 240 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2024-04-26T02:20:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd reporter subtree level = osd 2024-04-26T02:20:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd prime pg temp = true 2024-04-26T02:20:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2024-04-26T02:20:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T02:20:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T02:20:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T02:20:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T02:20:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T02:20:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T02:20:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: # 1m isn't quite enough 2024-04-26T02:20:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: mon_down_mkfs_grace = 2m 2024-04-26T02:20:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_filestore_osds = false 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: mon min osdmap epochs = 50 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune interval = 2 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune min = 15 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune txsize = 2 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: mon scrub interval = 300 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: paxos service trim min = 10 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout:[client] 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: rgw cache enabled = true 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable ops log = true 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable usage log = true 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: debug cephsqlite = 20 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: debug client = 20 2024-04-26T02:20:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-04-26T02:20:40.182 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.a] 2024-04-26T02:20:40.182 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.c] 2024-04-26T02:20:40.182 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.b] 2024-04-26T02:20:40.183 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-04-26T02:20:40.183 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-04-26T02:20:40.183 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T02:20:40.183 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-04-26T02:20:40.184 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.184 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-04-26T02:20:40.184 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.184 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-04-26T02:20:40.184 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.184 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-04-26T02:20:40.184 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-04-26T02:20:40.184 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.184 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-04-26T02:20:40.184 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-04-26T02:20:40.184 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.184 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T02:20:40.184 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-04-26T02:20:40.184 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-04-26T02:20:40.185 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.186 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T02:20:40.186 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.186 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-04-26T02:20:40.186 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.186 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-04-26T02:20:40.186 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.186 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-04-26T02:20:40.186 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-04-26T02:20:40.186 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-04-26T02:20:40.186 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = "[v1:172.21.15.87:6789],[v1:172.21.15.87:6790],[v1:172.21.15.122:6789]" 2024-04-26T02:20:40.186 INFO:teuthology.orchestra.run.smithi122.stdout: mon client directed command retry = 5 2024-04-26T02:20:40.186 INFO:teuthology.orchestra.run.smithi122.stdout: mon election default strategy = 1 2024-04-26T02:20:40.186 INFO:teuthology.orchestra.run.smithi122.stdout: ms bind msgr2 = False 2024-04-26T02:20:40.186 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject socket failures = 1500 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout: ms type = async 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = 6232cbab-48ed-4660-9de8-286f4c0e9a32 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-04-26T02:20:40.187 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-04-26T02:20:40.188 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug trim objects = true 2024-04-26T02:20:40.188 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.188 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-04-26T02:20:40.188 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-04-26T02:20:40.188 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.188 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-04-26T02:20:40.188 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.188 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-04-26T02:20:40.188 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.188 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-04-26T02:20:40.188 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.188 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-04-26T02:20:40.188 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-04-26T02:20:40.188 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.188 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-04-26T02:20:40.189 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-04-26T02:20:40.189 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-04-26T02:20:40.189 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore compression algorithm = zstd 2024-04-26T02:20:40.189 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore compression mode = aggressive 2024-04-26T02:20:40.189 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-04-26T02:20:40.189 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore zero block detection = True 2024-04-26T02:20:40.189 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 20 2024-04-26T02:20:40.189 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 20 2024-04-26T02:20:40.189 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-04-26T02:20:40.189 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-04-26T02:20:40.189 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 10 2024-04-26T02:20:40.189 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T02:20:40.189 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-04-26T02:20:40.189 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-04-26T02:20:40.189 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify cached snaps = True 2024-04-26T02:20:40.190 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify missing on start = True 2024-04-26T02:20:40.190 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-04-26T02:20:40.190 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock override recovery settings = True 2024-04-26T02:20:40.190 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-04-26T02:20:40.190 INFO:teuthology.orchestra.run.smithi122.stdout: osd op queue = debug_random 2024-04-26T02:20:40.190 INFO:teuthology.orchestra.run.smithi122.stdout: osd op queue cut off = debug_random 2024-04-26T02:20:40.190 INFO:teuthology.orchestra.run.smithi122.stdout: osd shutdown pgref assert = True 2024-04-26T02:20:40.190 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.190 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-04-26T02:20:40.190 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-04-26T02:20:40.190 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-04-26T02:20:40.190 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-04-26T02:20:40.190 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-04-26T02:20:40.190 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-04-26T02:20:40.190 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T02:20:40.191 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: mon min osdmap epochs = 50 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: mon osdmap full prune interval = 2 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: mon osdmap full prune min = 15 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: mon osdmap full prune txsize = 2 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: mon scrub interval = 300 2024-04-26T02:20:40.192 INFO:teuthology.orchestra.run.smithi122.stdout: paxos service trim min = 10 2024-04-26T02:20:40.193 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-26T02:20:40.193 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-04-26T02:20:40.193 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-04-26T02:20:40.193 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-04-26T02:20:40.193 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-04-26T02:20:40.193 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T02:20:40.193 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T02:20:40.193 INFO:teuthology.orchestra.run.smithi122.stdout: debug cephsqlite = 20 2024-04-26T02:20:40.193 INFO:teuthology.orchestra.run.smithi122.stdout: debug client = 20 2024-04-26T02:20:40.193 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-04-26T02:20:40.193 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-04-26T02:20:40.193 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.c] 2024-04-26T02:20:40.194 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.b] 2024-04-26T02:20:40.194 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-26T02:20:40.194 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-26T02:20:40.273 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-26T02:20:40.273 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:20:40.273 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-26T02:20:40.279 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:20:40.279 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-26T02:20:40.326 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi087.front.sepia.ceph.com 2024-04-26T02:20:40.327 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:20:40.327 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T02:20:40.327 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T02:20:40.391 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:20:40.391 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:20:40.439 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi122.front.sepia.ceph.com 2024-04-26T02:20:40.439 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:20:40.439 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T02:20:40.439 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T02:20:40.462 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:20:40.462 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:20:40.511 INFO:tasks.ceph:Setting up mon nodes... 2024-04-26T02:20:40.511 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-26T02:20:40.512 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-26T02:20:40.556 INFO:teuthology.orchestra.run.smithi087.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-26T02:20:40.559 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T02:20:40.602 INFO:teuthology.orchestra.run.smithi122.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T02:20:40.616 INFO:tasks.ceph:Setting up mds nodes... 2024-04-26T02:20:40.616 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-26T02:20:40.616 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T02:20:40.650 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-26T02:20:40.660 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-26T02:20:40.692 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-26T02:20:40.703 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-26T02:20:40.703 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-26T02:20:40.703 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-26T02:20:40.715 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T02:20:40.715 INFO:tasks.ceph:role: osd.0 2024-04-26T02:20:40.715 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com 2024-04-26T02:20:40.715 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T02:20:40.777 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T02:20:40.777 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T02:20:40.777 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T02:20:40.777 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T02:20:40.777 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T02:20:40.777 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-04-26T02:20:40.777 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T02:20:40.778 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T02:20:40.778 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T02:20:40.778 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T02:20:41.227 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-04-26T02:20:41.229 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-04-26T02:20:41.230 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-26T02:20:41.325 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-26T02:20:41.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:20:41.336 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T02:20:41.336 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-26T02:20:41.389 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T02:20:41.389 INFO:tasks.ceph:role: osd.1 2024-04-26T02:20:41.390 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com 2024-04-26T02:20:41.390 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T02:20:41.454 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T02:20:41.454 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T02:20:41.454 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T02:20:41.454 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T02:20:41.454 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T02:20:41.454 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-04-26T02:20:41.454 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T02:20:41.454 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T02:20:41.454 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T02:20:41.454 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T02:20:41.926 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-04-26T02:20:41.928 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-04-26T02:20:41.928 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-26T02:20:42.094 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-26T02:20:42.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:20:42.105 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T02:20:42.106 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-26T02:20:42.161 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T02:20:42.161 INFO:tasks.ceph:role: osd.2 2024-04-26T02:20:42.162 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com 2024-04-26T02:20:42.162 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T02:20:42.226 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T02:20:42.226 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T02:20:42.226 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T02:20:42.226 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T02:20:42.227 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T02:20:42.227 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-04-26T02:20:42.227 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T02:20:42.227 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T02:20:42.227 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T02:20:42.227 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T02:20:42.705 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-04-26T02:20:42.707 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-04-26T02:20:42.708 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-26T02:20:42.866 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-26T02:20:42.877 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T02:20:42.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:20:42.878 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-26T02:20:42.934 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T02:20:42.934 INFO:tasks.ceph:role: osd.3 2024-04-26T02:20:42.934 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi087.front.sepia.ceph.com 2024-04-26T02:20:42.934 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-26T02:20:42.998 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T02:20:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T02:20:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T02:20:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T02:20:42.998 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T02:20:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-04-26T02:20:42.998 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T02:20:42.998 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T02:20:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T02:20:42.999 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T02:20:43.426 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-04-26T02:20:43.428 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-04-26T02:20:43.428 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-26T02:20:43.524 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-26T02:20:43.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:20:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T02:20:43.536 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:20:43.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:43.649+0000 7f5519c267c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-26T02:20:43.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:43.653+0000 7f5519c267c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-26T02:20:43.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:43.653+0000 7f5519c267c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T02:20:43.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:43.653+0000 7f5519c267c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T02:20:43.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:43.653+0000 7f5519c267c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T02:20:43.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:43.661+0000 7f5519c267c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-26T02:20:46.623 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-26T02:20:46.636 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:20:46.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:46.717+0000 7fa32ef3f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-26T02:20:46.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:46.717+0000 7fa32ef3f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-26T02:20:46.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:46.717+0000 7fa32ef3f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T02:20:46.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:46.717+0000 7fa32ef3f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T02:20:46.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:46.717+0000 7fa32ef3f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T02:20:46.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:46.717+0000 7fa32ef3f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-26T02:20:49.703 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-26T02:20:49.758 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:20:49.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:49.845+0000 7fc7cc88b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-26T02:20:49.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:49.845+0000 7fc7cc88b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-26T02:20:49.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:49.845+0000 7fc7cc88b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T02:20:49.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:49.845+0000 7fc7cc88b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T02:20:49.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:49.845+0000 7fc7cc88b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T02:20:49.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:49.849+0000 7fc7cc88b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-26T02:20:52.814 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-26T02:20:52.871 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:20:52.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:52.953+0000 7f9f42e2c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-26T02:20:52.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:52.953+0000 7f9f42e2c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-26T02:20:52.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:52.953+0000 7f9f42e2c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T02:20:52.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:52.953+0000 7f9f42e2c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T02:20:52.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:52.953+0000 7f9f42e2c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T02:20:52.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:20:52.957+0000 7f9f42e2c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-26T02:20:55.938 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-26T02:20:55.993 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-26T02:20:56.007 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-26T02:20:56.007 INFO:tasks.ceph:role: osd.4 2024-04-26T02:20:56.008 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com 2024-04-26T02:20:56.008 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T02:20:56.070 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T02:20:56.071 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T02:20:56.071 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T02:20:56.071 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T02:20:56.071 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T02:20:56.071 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-04-26T02:20:56.071 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T02:20:56.071 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T02:20:56.071 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T02:20:56.071 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T02:20:56.519 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-04-26T02:20:56.521 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-04-26T02:20:56.521 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-26T02:20:56.657 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-26T02:20:56.667 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T02:20:56.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:20:56.669 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-26T02:20:56.722 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-26T02:20:56.722 INFO:tasks.ceph:role: osd.5 2024-04-26T02:20:56.722 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com 2024-04-26T02:20:56.722 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T02:20:56.786 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T02:20:56.786 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T02:20:56.787 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T02:20:56.787 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T02:20:56.787 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T02:20:56.787 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-04-26T02:20:56.787 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T02:20:56.787 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T02:20:56.787 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T02:20:56.787 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T02:20:57.218 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-04-26T02:20:57.220 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-04-26T02:20:57.220 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-26T02:20:57.284 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-26T02:20:57.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:20:57.336 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T02:20:57.337 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-26T02:20:57.390 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-26T02:20:57.390 INFO:tasks.ceph:role: osd.6 2024-04-26T02:20:57.390 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com 2024-04-26T02:20:57.390 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T02:20:57.454 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T02:20:57.454 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T02:20:57.454 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T02:20:57.454 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T02:20:57.454 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T02:20:57.454 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-04-26T02:20:57.454 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T02:20:57.454 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T02:20:57.455 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T02:20:57.455 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T02:20:57.900 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-04-26T02:20:57.902 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-04-26T02:20:57.902 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-26T02:20:57.997 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-26T02:20:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:20:58.007 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T02:20:58.007 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-26T02:20:58.060 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-26T02:20:58.060 INFO:tasks.ceph:role: osd.7 2024-04-26T02:20:58.060 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi122.front.sepia.ceph.com 2024-04-26T02:20:58.060 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-26T02:20:58.123 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T02:20:58.123 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T02:20:58.123 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T02:20:58.124 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T02:20:58.124 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T02:20:58.124 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-04-26T02:20:58.124 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T02:20:58.124 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T02:20:58.124 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T02:20:58.124 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T02:20:58.588 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-04-26T02:20:58.590 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-04-26T02:20:58.591 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-26T02:20:58.715 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-26T02:20:58.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:20:58.727 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T02:20:58.727 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:20:58.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:20:58.844+0000 7faf0f2d77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-26T02:20:58.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:20:58.844+0000 7faf0f2d77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-26T02:20:58.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:20:58.844+0000 7faf0f2d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T02:20:58.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:20:58.844+0000 7faf0f2d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T02:20:58.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:20:58.844+0000 7faf0f2d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T02:20:58.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:20:58.852+0000 7faf0f2d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-26T02:21:01.835 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-26T02:21:01.848 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:21:01.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:01.928+0000 7f08e93977c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-26T02:21:01.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:01.928+0000 7f08e93977c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-26T02:21:01.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:01.928+0000 7f08e93977c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T02:21:01.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:01.928+0000 7f08e93977c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T02:21:01.929 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:01.928+0000 7f08e93977c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T02:21:01.930 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:01.928+0000 7f08e93977c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-26T02:21:04.893 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-26T02:21:04.947 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:21:05.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:05.028+0000 7f6d38ffa7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-26T02:21:05.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:05.032+0000 7f6d38ffa7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-26T02:21:05.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:05.032+0000 7f6d38ffa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-26T02:21:05.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:05.032+0000 7f6d38ffa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-26T02:21:05.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:05.032+0000 7f6d38ffa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-26T02:21:05.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:05.032+0000 7f6d38ffa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-26T02:21:08.013 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-26T02:21:08.067 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:21:08.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:08.148+0000 7f5338eb07c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-26T02:21:08.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:08.148+0000 7f5338eb07c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-26T02:21:08.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:08.148+0000 7f5338eb07c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-26T02:21:08.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:08.148+0000 7f5338eb07c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-26T02:21:08.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:08.148+0000 7f5338eb07c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-26T02:21:08.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:08.148+0000 7f5338eb07c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-26T02:21:11.134 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-26T02:21:11.191 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-26T02:21:11.192 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:21:11.192 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-26T02:21:11.206 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:21:11.206 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-26T02:21:11.261 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:21:11.261 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-26T02:21:11.317 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:21:11.317 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-26T02:21:11.373 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:21:11.373 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-26T02:21:11.429 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:21:11.429 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-26T02:21:11.443 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:21:11.443 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-26T02:21:11.499 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:21:11.499 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-26T02:21:11.554 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:21:11.554 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-26T02:21:11.610 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:21:11.610 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-26T02:21:11.666 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:21:11.666 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-26T02:21:11.672 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:21:11.672 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-26T02:21:11.714 INFO:tasks.ceph:Adding keys to all mons... 2024-04-26T02:21:11.715 DEBUG:teuthology.orchestra.run.smithi087:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T02:21:11.717 DEBUG:teuthology.orchestra.run.smithi122:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T02:21:11.760 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr.y] 2024-04-26T02:21:11.761 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQB4DytmwlleIRAAGrmrhLv4YsT08HhsyAvwlg== 2024-04-26T02:21:11.761 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.0] 2024-04-26T02:21:11.761 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQB7DytmZ7LyJhAAr9CtlbnFFXJmBKdR16Nyog== 2024-04-26T02:21:11.761 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.1] 2024-04-26T02:21:11.761 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQB+Dytma4rgKhAAbWvZGtfOdHCVOMQmspADog== 2024-04-26T02:21:11.761 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.2] 2024-04-26T02:21:11.761 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCBDytmiauVMhAAvXJnTPRrkBJEusNBrWdXGA== 2024-04-26T02:21:11.761 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.3] 2024-04-26T02:21:11.761 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCEDytmKQIGORAA0c5UYn4j89GBtUkLlZ07UQ== 2024-04-26T02:21:11.761 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr.x] 2024-04-26T02:21:11.761 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQB4DytmB4oLJBAAJLyaLrrQp0U0iYmA2TVmbQ== 2024-04-26T02:21:11.761 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.4] 2024-04-26T02:21:11.761 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCKDytmYzVpMhAAMPlJrj9ng3H8+RAPh6S7Rw== 2024-04-26T02:21:11.762 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.5] 2024-04-26T02:21:11.762 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCNDytm0xp+NxAAHx1Qvc3YqjKPeiCYXkUTjg== 2024-04-26T02:21:11.762 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.6] 2024-04-26T02:21:11.762 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCRDytmCi3yARAAZXe+HDB63dWu/+Vi8ES45g== 2024-04-26T02:21:11.762 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.7] 2024-04-26T02:21:11.762 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCUDytmORb7CBAA37YkG2qxje79DKhbw4SdHA== 2024-04-26T02:21:11.762 INFO:teuthology.orchestra.run.smithi087.stdout:[client.0] 2024-04-26T02:21:11.762 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQB4DytmdMn7JhAA3RS2CWxx6/5AOixSCxYu6Q== 2024-04-26T02:21:11.762 INFO:teuthology.orchestra.run.smithi087.stdout:[client.1] 2024-04-26T02:21:11.762 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQB4DytmcFRpKRAANdA4e8YVIoyDCPCD3fCbpA== 2024-04-26T02:21:11.768 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr.y] 2024-04-26T02:21:11.768 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQB4DytmwlleIRAAGrmrhLv4YsT08HhsyAvwlg== 2024-04-26T02:21:11.768 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.0] 2024-04-26T02:21:11.768 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQB7DytmZ7LyJhAAr9CtlbnFFXJmBKdR16Nyog== 2024-04-26T02:21:11.768 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.1] 2024-04-26T02:21:11.769 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQB+Dytma4rgKhAAbWvZGtfOdHCVOMQmspADog== 2024-04-26T02:21:11.769 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.2] 2024-04-26T02:21:11.769 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCBDytmiauVMhAAvXJnTPRrkBJEusNBrWdXGA== 2024-04-26T02:21:11.769 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.3] 2024-04-26T02:21:11.769 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCEDytmKQIGORAA0c5UYn4j89GBtUkLlZ07UQ== 2024-04-26T02:21:11.769 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr.x] 2024-04-26T02:21:11.769 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQB4DytmB4oLJBAAJLyaLrrQp0U0iYmA2TVmbQ== 2024-04-26T02:21:11.769 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.4] 2024-04-26T02:21:11.769 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCKDytmYzVpMhAAMPlJrj9ng3H8+RAPh6S7Rw== 2024-04-26T02:21:11.769 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.5] 2024-04-26T02:21:11.769 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCNDytm0xp+NxAAHx1Qvc3YqjKPeiCYXkUTjg== 2024-04-26T02:21:11.769 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.6] 2024-04-26T02:21:11.769 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCRDytmCi3yARAAZXe+HDB63dWu/+Vi8ES45g== 2024-04-26T02:21:11.769 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.7] 2024-04-26T02:21:11.769 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCUDytmORb7CBAA37YkG2qxje79DKhbw4SdHA== 2024-04-26T02:21:11.770 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-04-26T02:21:11.770 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQB4DytmdMn7JhAA3RS2CWxx6/5AOixSCxYu6Q== 2024-04-26T02:21:11.770 INFO:teuthology.orchestra.run.smithi122.stdout:[client.1] 2024-04-26T02:21:11.770 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQB4DytmcFRpKRAANdA4e8YVIoyDCPCD3fCbpA== 2024-04-26T02:21:11.771 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T02:21:11.807 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T02:21:11.859 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:11.861 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:11.902 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:11.943 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:11.985 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:12.027 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:12.070 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:12.115 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:12.155 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T02:21:12.199 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T02:21:12.241 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:12.283 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:12.327 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:12.367 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:12.408 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:12.451 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:12.496 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:12.535 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T02:21:12.576 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T02:21:12.620 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T02:21:12.663 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T02:21:12.707 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T02:21:12.754 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-26T02:21:12.755 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-26T02:21:12.805 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T02:21:12.897 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-26T02:21:12.907 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-26T02:21:12.959 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T02:21:13.050 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-26T02:21:13.061 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-26T02:21:13.076 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T02:21:13.174 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-26T02:21:13.185 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:21:13.187 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T02:21:13.230 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-26T02:21:13.230 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T02:21:13.231 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T02:21:13.235 INFO:tasks.ceph.mon.a:Started 2024-04-26T02:21:13.235 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T02:21:13.235 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T02:21:13.237 INFO:tasks.ceph.mon.c:Started 2024-04-26T02:21:13.237 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T02:21:13.238 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T02:21:13.276 INFO:tasks.ceph.mon.b:Started 2024-04-26T02:21:13.276 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-26T02:21:13.276 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-26T02:21:13.276 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-26T02:21:13.278 INFO:tasks.ceph.mgr.y:Started 2024-04-26T02:21:13.279 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-26T02:21:13.279 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-26T02:21:13.280 INFO:tasks.ceph.mgr.x:Started 2024-04-26T02:21:13.281 INFO:tasks.ceph:Setting crush tunables to default 2024-04-26T02:21:13.281 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd crush tunables default 2024-04-26T02:21:13.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:13.401+0000 7f220e1cf640 1 Processor -- start 2024-04-26T02:21:13.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:13.401+0000 7f220e1cf640 1 -- start start 2024-04-26T02:21:13.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:13.401+0000 7f220e1cf640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2208140240 con 0x7f22080b0000 2024-04-26T02:21:13.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:13.401+0000 7f220e1cf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2208072e70 con 0x7f22080b0840 2024-04-26T02:21:13.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:13.401+0000 7f220e1cf640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f220814e700 con 0x7f220813fdb0 2024-04-26T02:21:13.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:13.401+0000 7f22077fe640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f22080b0000 0x7f2208140760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:40180/0 (socket says 172.21.15.87:40180) 2024-04-26T02:21:13.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:13.401+0000 7f22077fe640 1 -- 172.21.15.87:0/1058465244 learned_addr learned my addr 172.21.15.87:0/1058465244 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:13.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:13.401+0000 7f21fffff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f220813fdb0 0x7f2208071160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:35794/0 (socket says 172.21.15.87:35794) 2024-04-26T02:21:16.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:16.401+0000 7f2205ffb640 1 -- 172.21.15.87:0/1058465244 >> v1:172.21.15.87:6790/0 conn(0x7f220813fdb0 legacy=0x7f2208071160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:16.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:16.401+0000 7f2205ffb640 1 -- 172.21.15.87:0/1058465244 >> v1:172.21.15.122:6789/0 conn(0x7f22080b0840 legacy=0x7f22080b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:16.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:16.401+0000 7f2205ffb640 1 -- 172.21.15.87:0/1058465244 >> v1:172.21.15.87:6789/0 conn(0x7f22080b0000 legacy=0x7f2208140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:16.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:16.401+0000 7f2205ffb640 1 -- 172.21.15.87:0/1058465244 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f220814f8e0 con 0x7f21e800b010 2024-04-26T02:21:16.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:16.401+0000 7f2205ffb640 1 -- 172.21.15.87:0/1058465244 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21e800fdb0 con 0x7f21e80072e0 2024-04-26T02:21:16.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:16.401+0000 7f2205ffb640 1 -- 172.21.15.87:0/1058465244 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21e8010fe0 con 0x7f21e80036f0 2024-04-26T02:21:18.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.649+0000 7f2206ffd640 1 -- 172.21.15.87:0/1058465244 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2403499636 0 0) 0x7f220814f8e0 con 0x7f21e800b010 2024-04-26T02:21:18.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.649+0000 7f2206ffd640 1 -- 172.21.15.87:0/1058465244 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21f4003530 con 0x7f21e800b010 2024-04-26T02:21:18.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.649+0000 7f2206ffd640 1 -- 172.21.15.87:0/1058465244 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2232317535 0 0) 0x7f21e800fdb0 con 0x7f21e80072e0 2024-04-26T02:21:18.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.649+0000 7f2206ffd640 1 -- 172.21.15.87:0/1058465244 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f220814f8e0 con 0x7f21e80072e0 2024-04-26T02:21:18.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.649+0000 7f2206ffd640 1 -- 172.21.15.87:0/1058465244 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1679638325 0 0) 0x7f21e8010fe0 con 0x7f21e80036f0 2024-04-26T02:21:18.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.649+0000 7f2206ffd640 1 -- 172.21.15.87:0/1058465244 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21e800fdb0 con 0x7f21e80036f0 2024-04-26T02:21:18.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.649+0000 7f2206ffd640 1 -- 172.21.15.87:0/1058465244 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3973764683 0 0) 0x7f21e800fdb0 con 0x7f21e80036f0 2024-04-26T02:21:18.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.649+0000 7f2206ffd640 1 -- 172.21.15.87:0/1058465244 >> v1:172.21.15.122:6789/0 conn(0x7f21e80072e0 legacy=0x7f21e8009770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:18.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.649+0000 7f2206ffd640 1 -- 172.21.15.87:0/1058465244 >> v1:172.21.15.87:6789/0 conn(0x7f21e800b010 legacy=0x7f21e800d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:18.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.649+0000 7f2206ffd640 1 -- 172.21.15.87:0/1058465244 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e80122d0 con 0x7f21e80036f0 2024-04-26T02:21:18.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.649+0000 7f2206ffd640 1 -- 172.21.15.87:0/1058465244 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f21f8001660 con 0x7f21e80036f0 2024-04-26T02:21:18.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.653+0000 7f2206ffd640 1 -- 172.21.15.87:0/1058465244 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f21f80018c0 con 0x7f21e80036f0 2024-04-26T02:21:18.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.653+0000 7f2206ffd640 1 -- 172.21.15.87:0/1058465244 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f21f8001cd0 con 0x7f21e80036f0 2024-04-26T02:21:18.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.653+0000 7f220e1cf640 1 -- 172.21.15.87:0/1058465244 >> v1:172.21.15.87:6790/0 conn(0x7f21e80036f0 legacy=0x7f21e8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:18.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.653+0000 7f220e1cf640 1 -- 172.21.15.87:0/1058465244 shutdown_connections 2024-04-26T02:21:18.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.653+0000 7f220e1cf640 1 -- 172.21.15.87:0/1058465244 >> 172.21.15.87:0/1058465244 conn(0x7f22080b5b70 msgr2=0x7f22080b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:18.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.653+0000 7f220e1cf640 1 -- 172.21.15.87:0/1058465244 shutdown_connections 2024-04-26T02:21:18.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.653+0000 7f220e1cf640 1 -- 172.21.15.87:0/1058465244 wait complete. 2024-04-26T02:21:18.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f220e1cf640 1 Processor -- start 2024-04-26T02:21:18.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f220e1cf640 1 -- start start 2024-04-26T02:21:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f220e1cf640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f220806e1f0 con 0x7f21e80036f0 2024-04-26T02:21:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f220e1cf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f220806c2c0 con 0x7f21e80072e0 2024-04-26T02:21:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f220e1cf640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f220806d6b0 con 0x7f21e800b010 2024-04-26T02:21:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f22077fe640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f21e80036f0 0x7f220806ce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:35364/0 (socket says 172.21.15.87:35364) 2024-04-26T02:21:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f22077fe640 1 -- 172.21.15.87:0/1320713724 learned_addr learned my addr 172.21.15.87:0/1320713724 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:18.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1909216000 0 0) 0x7f220806d6b0 con 0x7f21e800b010 2024-04-26T02:21:18.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21e8008e50 con 0x7f21e800b010 2024-04-26T02:21:18.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2251336928 0 0) 0x7f220806e1f0 con 0x7f21e80036f0 2024-04-26T02:21:18.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f220806d6b0 con 0x7f21e80036f0 2024-04-26T02:21:18.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3628423094 0 0) 0x7f220806d6b0 con 0x7f21e80036f0 2024-04-26T02:21:18.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f220806e1f0 con 0x7f21e80036f0 2024-04-26T02:21:18.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f21f8001c10 con 0x7f21e80036f0 2024-04-26T02:21:18.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2187139525 0 0) 0x7f220806e1f0 con 0x7f21e80036f0 2024-04-26T02:21:18.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 >> v1:172.21.15.87:6790/0 conn(0x7f21e800b010 legacy=0x7f220813b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:18.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 >> v1:172.21.15.122:6789/0 conn(0x7f21e80072e0 legacy=0x7f2208135e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:18.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220806d880 con 0x7f21e80036f0 2024-04-26T02:21:18.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f220e1cf640 1 -- 172.21.15.87:0/1320713724 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2208136530 con 0x7f21e80036f0 2024-04-26T02:21:18.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.657+0000 7f220e1cf640 1 -- 172.21.15.87:0/1320713724 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f220813bf60 con 0x7f21e80036f0 2024-04-26T02:21:18.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.661+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f21f80018c0 con 0x7f21e80036f0 2024-04-26T02:21:18.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.661+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f21f8004aa0 con 0x7f21e80036f0 2024-04-26T02:21:18.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.661+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f21f8004fd0 con 0x7f21e80036f0 2024-04-26T02:21:18.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.661+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3734756610 0 0) 0x7f21f80066c0 con 0x7f21e80036f0 2024-04-26T02:21:18.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.661+0000 7f220e1cf640 1 -- 172.21.15.87:0/1320713724 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21ec005090 con 0x7f21e80036f0 2024-04-26T02:21:18.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.685+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f21f80068c0 con 0x7f21e80036f0 2024-04-26T02:21:18.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.753+0000 7f220e1cf640 1 -- 172.21.15.87:0/1320713724 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f21ec005380 con 0x7f21e80036f0 2024-04-26T02:21:18.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.757+0000 7f2204ff9640 1 -- 172.21.15.87:0/1320713724 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (unknown 2277712955 0 0) 0x7f21f80056a0 con 0x7f21e80036f0 2024-04-26T02:21:18.754 INFO:teuthology.orchestra.run.smithi087.stderr:adjusted tunables profile to default 2024-04-26T02:21:18.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.757+0000 7f220e1cf640 1 -- 172.21.15.87:0/1320713724 >> v1:172.21.15.87:6789/0 conn(0x7f21e80036f0 legacy=0x7f220806ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:18.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.757+0000 7f220e1cf640 1 -- 172.21.15.87:0/1320713724 shutdown_connections 2024-04-26T02:21:18.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.757+0000 7f220e1cf640 1 -- 172.21.15.87:0/1320713724 >> 172.21.15.87:0/1320713724 conn(0x7f22080b5b70 msgr2=0x7f220813e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:18.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.757+0000 7f220e1cf640 1 -- 172.21.15.87:0/1320713724 shutdown_connections 2024-04-26T02:21:18.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:18.757+0000 7f220e1cf640 1 -- 172.21.15.87:0/1320713724 wait complete. 2024-04-26T02:21:18.778 INFO:tasks.ceph:check_enable_crimson: False 2024-04-26T02:21:18.778 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-26T02:21:18.778 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:21:18.778 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-26T02:21:18.789 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:21:18.789 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-26T02:21:18.848 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:21:18.848 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-26T02:21:18.878 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:18.881+0000 7f024dae3180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T02:21:18.889 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:21:18.889 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-26T02:21:18.909 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:18.908+0000 7fc960988180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T02:21:18.945 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:21:18.945 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-26T02:21:18.958 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:21:18.958 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-26T02:21:19.013 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:21:19.013 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-26T02:21:19.066 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-26T02:21:19.067 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-26T02:21:19.122 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new 6ea27580-6f9b-4e32-813f-fb8923bb6180 0 2024-04-26T02:21:19.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72d4a20640 1 Processor -- start 2024-04-26T02:21:19.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72d4a20640 1 -- start start 2024-04-26T02:21:19.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72d4a20640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72d00afdb0 con 0x7f72d013d770 2024-04-26T02:21:19.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72d4a20640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72d00aead0 con 0x7f72d00aedd0 2024-04-26T02:21:19.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72d4a20640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72d00afa40 con 0x7f72d00670b0 2024-04-26T02:21:19.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72ced9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f72d00aedd0 0x7f72d014b760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45098/0 (socket says 172.21.15.122:45098) 2024-04-26T02:21:19.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72ced9d640 1 -- 172.21.15.122:0/498515217 learned_addr learned my addr 172.21.15.122:0/498515217 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:19.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72cd59a640 1 -- 172.21.15.122:0/498515217 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 524287355 0 0) 0x7f72d00aead0 con 0x7f72d00aedd0 2024-04-26T02:21:19.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72cd59a640 1 -- 172.21.15.122:0/498515217 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72b8003530 con 0x7f72d00aedd0 2024-04-26T02:21:19.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72cd59a640 1 -- 172.21.15.122:0/498515217 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1444171988 0 0) 0x7f72b8003530 con 0x7f72d00aedd0 2024-04-26T02:21:19.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72cd59a640 1 -- 172.21.15.122:0/498515217 >> v1:172.21.15.87:6790/0 conn(0x7f72d00670b0 legacy=0x7f72d013d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72cd59a640 1 -- 172.21.15.122:0/498515217 >> v1:172.21.15.87:6789/0 conn(0x7f72d013d770 legacy=0x7f72d01433c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72cd59a640 1 -- 172.21.15.122:0/498515217 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d0143ad0 con 0x7f72d00aedd0 2024-04-26T02:21:19.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72cd59a640 1 -- 172.21.15.122:0/498515217 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f72c80019a0 con 0x7f72d00aedd0 2024-04-26T02:21:19.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72cd59a640 1 -- 172.21.15.122:0/498515217 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72c8001d00 con 0x7f72d00aedd0 2024-04-26T02:21:19.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72cd59a640 1 -- 172.21.15.122:0/498515217 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f72c80020d0 con 0x7f72d00aedd0 2024-04-26T02:21:19.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72d4a20640 1 -- 172.21.15.122:0/498515217 >> v1:172.21.15.122:6789/0 conn(0x7f72d00aedd0 legacy=0x7f72d014b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72d4a20640 1 -- 172.21.15.122:0/498515217 shutdown_connections 2024-04-26T02:21:19.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72d4a20640 1 -- 172.21.15.122:0/498515217 >> 172.21.15.122:0/498515217 conn(0x7f72d00b4180 msgr2=0x7f72d00b4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:19.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72d4a20640 1 -- 172.21.15.122:0/498515217 shutdown_connections 2024-04-26T02:21:19.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72d4a20640 1 -- 172.21.15.122:0/498515217 wait complete. 2024-04-26T02:21:19.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72d4a20640 1 Processor -- start 2024-04-26T02:21:19.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.292+0000 7f72d4a20640 1 -- start start 2024-04-26T02:21:19.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72d4a20640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72d00a9680 con 0x7f72d013d770 2024-04-26T02:21:19.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72d4a20640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72d01d9ea0 con 0x7f72d00670b0 2024-04-26T02:21:19.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72d4a20640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72d00a98f0 con 0x7f72d00aedd0 2024-04-26T02:21:19.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72ce59c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f72d00670b0 0x7f72d013ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45110/0 (socket says 172.21.15.122:45110) 2024-04-26T02:21:19.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72ce59c640 1 -- 172.21.15.122:0/1918982681 learned_addr learned my addr 172.21.15.122:0/1918982681 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:19.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2709938283 0 0) 0x7f72d01d9ea0 con 0x7f72d00670b0 2024-04-26T02:21:19.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72b0003530 con 0x7f72d00670b0 2024-04-26T02:21:19.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2561244574 0 0) 0x7f72b0003530 con 0x7f72d00670b0 2024-04-26T02:21:19.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72d01d9ea0 con 0x7f72d00670b0 2024-04-26T02:21:19.296 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1728076836 0 0) 0x7f72d00a98f0 con 0x7f72d00aedd0 2024-04-26T02:21:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72b0003530 con 0x7f72d00aedd0 2024-04-26T02:21:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3906971958 0 0) 0x7f72d00a9680 con 0x7f72d013d770 2024-04-26T02:21:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72d00a98f0 con 0x7f72d013d770 2024-04-26T02:21:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f72c0001980 con 0x7f72d00670b0 2024-04-26T02:21:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 <== mon.1 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2502671297 0 0) 0x7f72d01d9ea0 con 0x7f72d00670b0 2024-04-26T02:21:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 >> v1:172.21.15.87:6790/0 conn(0x7f72d00aedd0 legacy=0x7f72d01d9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 >> v1:172.21.15.87:6789/0 conn(0x7f72d013d770 legacy=0x7f72d01d9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d00a9bd0 con 0x7f72d00670b0 2024-04-26T02:21:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72d4a20640 1 -- 172.21.15.122:0/1918982681 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f72d01427b0 con 0x7f72d00670b0 2024-04-26T02:21:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72d4a20640 1 -- 172.21.15.122:0/1918982681 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f72d01406c0 con 0x7f72d00670b0 2024-04-26T02:21:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 <== mon.1 v1:172.21.15.122:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72c00038d0 con 0x7f72d00670b0 2024-04-26T02:21:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 <== mon.1 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f72c0003ca0 con 0x7f72d00670b0 2024-04-26T02:21:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 <== mon.1 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f72c0004250 con 0x7f72d00670b0 2024-04-26T02:21:19.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 <== mon.1 v1:172.21.15.122:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (unknown 3208616596 0 0) 0x7f72c0006a80 con 0x7f72d00670b0 2024-04-26T02:21:19.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.296+0000 7f72d4a20640 1 -- 172.21.15.122:0/1918982681 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72b4005090 con 0x7f72d00670b0 2024-04-26T02:21:19.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.300+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 <== mon.1 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f72c0005ba0 con 0x7f72d00670b0 2024-04-26T02:21:19.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.368+0000 7f72d4a20640 1 -- 172.21.15.122:0/1918982681 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6ea27580-6f9b-4e32-813f-fb8923bb6180", "id": 0} v 0) -- 0x7f72b4005380 con 0x7f72d00670b0 2024-04-26T02:21:19.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.372+0000 7f72beffd640 1 -- 172.21.15.122:0/1918982681 <== mon.1 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6ea27580-6f9b-4e32-813f-fb8923bb6180", "id": 0}]=0 v3) ==== 112+0+2 (unknown 3042079955 0 732707515) 0x7f72c0005f20 con 0x7f72d00670b0 2024-04-26T02:21:19.373 INFO:teuthology.orchestra.run.smithi122.stdout:0 2024-04-26T02:21:19.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.376+0000 7f72d4a20640 1 -- 172.21.15.122:0/1918982681 >> v1:172.21.15.122:6789/0 conn(0x7f72d00670b0 legacy=0x7f72d013ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.376+0000 7f72d4a20640 1 -- 172.21.15.122:0/1918982681 shutdown_connections 2024-04-26T02:21:19.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.376+0000 7f72d4a20640 1 -- 172.21.15.122:0/1918982681 >> 172.21.15.122:0/1918982681 conn(0x7f72d00b4180 msgr2=0x7f72d0136b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:19.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.376+0000 7f72d4a20640 1 -- 172.21.15.122:0/1918982681 shutdown_connections 2024-04-26T02:21:19.374 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.376+0000 7f72d4a20640 1 -- 172.21.15.122:0/1918982681 wait complete. 2024-04-26T02:21:19.390 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new b8afe0d9-5e03-4ae1-b91d-54b1b7071d1f 1 2024-04-26T02:21:19.421 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:19.421+0000 7f024dae3180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T02:21:19.427 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:19.428+0000 7fc960988180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T02:21:19.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b47fd0640 1 Processor -- start 2024-04-26T02:21:19.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b47fd0640 1 -- start start 2024-04-26T02:21:19.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b47fd0640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b4013f280 con 0x7f0b400703e0 2024-04-26T02:21:19.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b47fd0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b40072620 con 0x7f0b400707c0 2024-04-26T02:21:19.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b47fd0640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b40072900 con 0x7f0b401389e0 2024-04-26T02:21:19.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b4556b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0b400707c0 0x7f0b4013f760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45112/0 (socket says 172.21.15.122:45112) 2024-04-26T02:21:19.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b4556b640 1 -- 172.21.15.122:0/98594973 learned_addr learned my addr 172.21.15.122:0/98594973 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:19.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b44d6a640 1 -- 172.21.15.122:0/98594973 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2278667031 0 0) 0x7f0b40072620 con 0x7f0b400707c0 2024-04-26T02:21:19.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b44d6a640 1 -- 172.21.15.122:0/98594973 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b2c003530 con 0x7f0b400707c0 2024-04-26T02:21:19.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b44d6a640 1 -- 172.21.15.122:0/98594973 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3603655723 0 0) 0x7f0b2c003530 con 0x7f0b400707c0 2024-04-26T02:21:19.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b44d6a640 1 -- 172.21.15.122:0/98594973 >> v1:172.21.15.87:6790/0 conn(0x7f0b401389e0 legacy=0x7f0b4014a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b44d6a640 1 -- 172.21.15.122:0/98594973 >> v1:172.21.15.87:6789/0 conn(0x7f0b400703e0 legacy=0x7f0b400af410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b44d6a640 1 -- 172.21.15.122:0/98594973 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b400715a0 con 0x7f0b400707c0 2024-04-26T02:21:19.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b44d6a640 1 -- 172.21.15.122:0/98594973 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f0b30001620 con 0x7f0b400707c0 2024-04-26T02:21:19.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b44d6a640 1 -- 172.21.15.122:0/98594973 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0b30001980 con 0x7f0b400707c0 2024-04-26T02:21:19.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b44d6a640 1 -- 172.21.15.122:0/98594973 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f0b30001d50 con 0x7f0b400707c0 2024-04-26T02:21:19.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b47fd0640 1 -- 172.21.15.122:0/98594973 >> v1:172.21.15.122:6789/0 conn(0x7f0b400707c0 legacy=0x7f0b4013f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b47fd0640 1 -- 172.21.15.122:0/98594973 shutdown_connections 2024-04-26T02:21:19.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b47fd0640 1 -- 172.21.15.122:0/98594973 >> 172.21.15.122:0/98594973 conn(0x7f0b400b4a10 msgr2=0x7f0b400b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:19.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b47fd0640 1 -- 172.21.15.122:0/98594973 shutdown_connections 2024-04-26T02:21:19.510 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b47fd0640 1 -- 172.21.15.122:0/98594973 wait complete. 2024-04-26T02:21:19.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b47fd0640 1 Processor -- start 2024-04-26T02:21:19.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.508+0000 7f0b47fd0640 1 -- start start 2024-04-26T02:21:19.513 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b47fd0640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b401359e0 con 0x7f0b401389e0 2024-04-26T02:21:19.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b47fd0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b4013bc20 con 0x7f0b400707c0 2024-04-26T02:21:19.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b47fd0640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b401302f0 con 0x7f0b400703e0 2024-04-26T02:21:19.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b4556b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0b400707c0 0x7f0b401d8350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45120/0 (socket says 172.21.15.122:45120) 2024-04-26T02:21:19.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b4556b640 1 -- 172.21.15.122:0/3139542355 learned_addr learned my addr 172.21.15.122:0/3139542355 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:19.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 194089154 0 0) 0x7f0b4013bc20 con 0x7f0b400707c0 2024-04-26T02:21:19.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b24003530 con 0x7f0b400707c0 2024-04-26T02:21:19.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 182367499 0 0) 0x7f0b24003530 con 0x7f0b400707c0 2024-04-26T02:21:19.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b4013bc20 con 0x7f0b400707c0 2024-04-26T02:21:19.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f0b300015e0 con 0x7f0b400707c0 2024-04-26T02:21:19.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3604519474 0 0) 0x7f0b401302f0 con 0x7f0b400703e0 2024-04-26T02:21:19.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b24003530 con 0x7f0b400703e0 2024-04-26T02:21:19.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 <== mon.1 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3904664339 0 0) 0x7f0b4013bc20 con 0x7f0b400707c0 2024-04-26T02:21:19.514 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 >> v1:172.21.15.87:6790/0 conn(0x7f0b400703e0 legacy=0x7f0b4012fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 >> v1:172.21.15.87:6789/0 conn(0x7f0b401389e0 legacy=0x7f0b401d8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b401304c0 con 0x7f0b400707c0 2024-04-26T02:21:19.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b47fd0640 1 -- 172.21.15.122:0/3139542355 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0b40133d20 con 0x7f0b400707c0 2024-04-26T02:21:19.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b47fd0640 1 -- 172.21.15.122:0/3139542355 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0b40142630 con 0x7f0b400707c0 2024-04-26T02:21:19.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 <== mon.1 v1:172.21.15.122:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0b30004dc0 con 0x7f0b400707c0 2024-04-26T02:21:19.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 <== mon.1 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f0b30005190 con 0x7f0b400707c0 2024-04-26T02:21:19.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 <== mon.1 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0b300056c0 con 0x7f0b400707c0 2024-04-26T02:21:19.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 <== mon.1 v1:172.21.15.122:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (unknown 3226915885 0 0) 0x7f0b30006e10 con 0x7f0b400707c0 2024-04-26T02:21:19.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b47fd0640 1 -- 172.21.15.122:0/3139542355 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b28005090 con 0x7f0b400707c0 2024-04-26T02:21:19.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.512+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 <== mon.1 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0b30005be0 con 0x7f0b400707c0 2024-04-26T02:21:19.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.580+0000 7f0b47fd0640 1 -- 172.21.15.122:0/3139542355 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b8afe0d9-5e03-4ae1-b91d-54b1b7071d1f", "id": 1} v 0) -- 0x7f0b28005380 con 0x7f0b400707c0 2024-04-26T02:21:19.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.584+0000 7f0b367fc640 1 -- 172.21.15.122:0/3139542355 <== mon.1 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b8afe0d9-5e03-4ae1-b91d-54b1b7071d1f", "id": 1}]=0 v4) ==== 112+0+2 (unknown 1215700776 0 940483276) 0x7f0b30005e40 con 0x7f0b400707c0 2024-04-26T02:21:19.586 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-04-26T02:21:19.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.588+0000 7f0b47fd0640 1 -- 172.21.15.122:0/3139542355 >> v1:172.21.15.122:6789/0 conn(0x7f0b400707c0 legacy=0x7f0b401d8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.588+0000 7f0b47fd0640 1 -- 172.21.15.122:0/3139542355 shutdown_connections 2024-04-26T02:21:19.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.588+0000 7f0b47fd0640 1 -- 172.21.15.122:0/3139542355 >> 172.21.15.122:0/3139542355 conn(0x7f0b400b4a10 msgr2=0x7f0b40132950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:19.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.588+0000 7f0b47fd0640 1 -- 172.21.15.122:0/3139542355 shutdown_connections 2024-04-26T02:21:19.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.588+0000 7f0b47fd0640 1 -- 172.21.15.122:0/3139542355 wait complete. 2024-04-26T02:21:19.602 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new d3a9234e-ef70-421d-b5ba-121f4f5cdcc6 2 2024-04-26T02:21:19.625 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:19.624+0000 7fc960988180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T02:21:19.628 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:19.629+0000 7f024dae3180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T02:21:19.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.740+0000 7fb5e66f8640 1 Processor -- start 2024-04-26T02:21:19.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.740+0000 7fb5e66f8640 1 -- start start 2024-04-26T02:21:19.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.740+0000 7fb5e66f8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5e00a8ce0 con 0x7fb5e00afea0 2024-04-26T02:21:19.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.740+0000 7fb5e66f8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5e00a8fc0 con 0x7fb5e00724d0 2024-04-26T02:21:19.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.740+0000 7fb5e66f8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5e00b0880 con 0x7fb5e013d580 2024-04-26T02:21:19.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.740+0000 7fb5dffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb5e00724d0 0x7fb5e0143550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45128/0 (socket says 172.21.15.122:45128) 2024-04-26T02:21:19.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.740+0000 7fb5dffff640 1 -- 172.21.15.122:0/80526958 learned_addr learned my addr 172.21.15.122:0/80526958 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:19.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.740+0000 7fb5deffd640 1 -- 172.21.15.122:0/80526958 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3877900409 0 0) 0x7fb5e00a8fc0 con 0x7fb5e00724d0 2024-04-26T02:21:19.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.740+0000 7fb5deffd640 1 -- 172.21.15.122:0/80526958 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5cc003530 con 0x7fb5e00724d0 2024-04-26T02:21:19.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5deffd640 1 -- 172.21.15.122:0/80526958 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2713307509 0 0) 0x7fb5cc003530 con 0x7fb5e00724d0 2024-04-26T02:21:19.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5deffd640 1 -- 172.21.15.122:0/80526958 >> v1:172.21.15.87:6790/0 conn(0x7fb5e013d580 legacy=0x7fb5e014b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5deffd640 1 -- 172.21.15.122:0/80526958 >> v1:172.21.15.87:6789/0 conn(0x7fb5e00afea0 legacy=0x7fb5e013ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5deffd640 1 -- 172.21.15.122:0/80526958 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5e00af980 con 0x7fb5e00724d0 2024-04-26T02:21:19.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5deffd640 1 -- 172.21.15.122:0/80526958 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb5d8001620 con 0x7fb5e00724d0 2024-04-26T02:21:19.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5deffd640 1 -- 172.21.15.122:0/80526958 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb5d8001910 con 0x7fb5e00724d0 2024-04-26T02:21:19.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5deffd640 1 -- 172.21.15.122:0/80526958 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb5d8001ca0 con 0x7fb5e00724d0 2024-04-26T02:21:19.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5e66f8640 1 -- 172.21.15.122:0/80526958 >> v1:172.21.15.122:6789/0 conn(0x7fb5e00724d0 legacy=0x7fb5e0143550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5e66f8640 1 -- 172.21.15.122:0/80526958 shutdown_connections 2024-04-26T02:21:19.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5e66f8640 1 -- 172.21.15.122:0/80526958 >> 172.21.15.122:0/80526958 conn(0x7fb5e00b4a10 msgr2=0x7fb5e00b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:19.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5e66f8640 1 -- 172.21.15.122:0/80526958 shutdown_connections 2024-04-26T02:21:19.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5e66f8640 1 -- 172.21.15.122:0/80526958 wait complete. 2024-04-26T02:21:19.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5e66f8640 1 Processor -- start 2024-04-26T02:21:19.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5e66f8640 1 -- start start 2024-04-26T02:21:19.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5e66f8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5e0134580 con 0x7fb5e00afea0 2024-04-26T02:21:19.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5e66f8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5e013c030 con 0x7fb5e00724d0 2024-04-26T02:21:19.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5e66f8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5e013c4c0 con 0x7fb5e013d580 2024-04-26T02:21:19.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb5e00724d0 0x7fb5e01d8860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45136/0 (socket says 172.21.15.122:45136) 2024-04-26T02:21:19.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dffff640 1 -- 172.21.15.122:0/476970563 learned_addr learned my addr 172.21.15.122:0/476970563 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:19.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3682564678 0 0) 0x7fb5e013c030 con 0x7fb5e00724d0 2024-04-26T02:21:19.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5c0003530 con 0x7fb5e00724d0 2024-04-26T02:21:19.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 302924913 0 0) 0x7fb5c0003530 con 0x7fb5e00724d0 2024-04-26T02:21:19.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb5e013c030 con 0x7fb5e00724d0 2024-04-26T02:21:19.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb5d80015e0 con 0x7fb5e00724d0 2024-04-26T02:21:19.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2924662514 0 0) 0x7fb5e013c4c0 con 0x7fb5e013d580 2024-04-26T02:21:19.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5c0003530 con 0x7fb5e013d580 2024-04-26T02:21:19.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4066723142 0 0) 0x7fb5e0134580 con 0x7fb5e00afea0 2024-04-26T02:21:19.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5e013c4c0 con 0x7fb5e00afea0 2024-04-26T02:21:19.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 <== mon.1 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3749402973 0 0) 0x7fb5e013c030 con 0x7fb5e00724d0 2024-04-26T02:21:19.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 >> v1:172.21.15.87:6790/0 conn(0x7fb5e013d580 legacy=0x7fb5e0133e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 >> v1:172.21.15.87:6789/0 conn(0x7fb5e00afea0 legacy=0x7fb5e0133760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5e013c690 con 0x7fb5e00724d0 2024-04-26T02:21:19.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5e66f8640 1 -- 172.21.15.122:0/476970563 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb5e0140570 con 0x7fb5e00724d0 2024-04-26T02:21:19.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5e66f8640 1 -- 172.21.15.122:0/476970563 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb5e0139e80 con 0x7fb5e00724d0 2024-04-26T02:21:19.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 <== mon.1 v1:172.21.15.122:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb5d8004cd0 con 0x7fb5e00724d0 2024-04-26T02:21:19.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 <== mon.1 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb5d8004f00 con 0x7fb5e00724d0 2024-04-26T02:21:19.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 <== mon.1 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb5d8005430 con 0x7fb5e00724d0 2024-04-26T02:21:19.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 <== mon.1 v1:172.21.15.122:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (unknown 2866064202 0 0) 0x7fb5d8006890 con 0x7fb5e00724d0 2024-04-26T02:21:19.748 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.744+0000 7fb5e66f8640 1 -- 172.21.15.122:0/476970563 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5c4005090 con 0x7fb5e00724d0 2024-04-26T02:21:19.749 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.748+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 <== mon.1 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb5d8006ac0 con 0x7fb5e00724d0 2024-04-26T02:21:19.815 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.816+0000 7fb5e66f8640 1 -- 172.21.15.122:0/476970563 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d3a9234e-ef70-421d-b5ba-121f4f5cdcc6", "id": 2} v 0) -- 0x7fb5c4005380 con 0x7fb5e00724d0 2024-04-26T02:21:19.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.820+0000 7fb5dcff9640 1 -- 172.21.15.122:0/476970563 <== mon.1 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d3a9234e-ef70-421d-b5ba-121f4f5cdcc6", "id": 2}]=0 v5) ==== 112+0+2 (unknown 1103794006 0 216599125) 0x7fb5d80057f0 con 0x7fb5e00724d0 2024-04-26T02:21:19.821 INFO:teuthology.orchestra.run.smithi122.stdout:2 2024-04-26T02:21:19.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.824+0000 7fb5e66f8640 1 -- 172.21.15.122:0/476970563 >> v1:172.21.15.122:6789/0 conn(0x7fb5e00724d0 legacy=0x7fb5e01d8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.824+0000 7fb5e66f8640 1 -- 172.21.15.122:0/476970563 shutdown_connections 2024-04-26T02:21:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.824+0000 7fb5e66f8640 1 -- 172.21.15.122:0/476970563 >> 172.21.15.122:0/476970563 conn(0x7fb5e00b4a10 msgr2=0x7fb5e00b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.824+0000 7fb5e66f8640 1 -- 172.21.15.122:0/476970563 shutdown_connections 2024-04-26T02:21:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.824+0000 7fb5e66f8640 1 -- 172.21.15.122:0/476970563 wait complete. 2024-04-26T02:21:19.837 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new bfc148fb-9fd7-4a94-8287-52a4f38d7100 3 2024-04-26T02:21:19.952 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.952+0000 7f2856cd4640 1 Processor -- start 2024-04-26T02:21:19.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.952+0000 7f2856cd4640 1 -- start start 2024-04-26T02:21:19.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.952+0000 7f2856cd4640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28500a8ce0 con 0x7f285013cf90 2024-04-26T02:21:19.953 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.952+0000 7f2856cd4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28500a8fc0 con 0x7f28501498e0 2024-04-26T02:21:19.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.952+0000 7f2856cd4640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28500b0880 con 0x7f28500724d0 2024-04-26T02:21:19.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.952+0000 7f2855271640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f28501498e0 0x7f285014bcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45146/0 (socket says 172.21.15.122:45146) 2024-04-26T02:21:19.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.952+0000 7f2855271640 1 -- 172.21.15.122:0/588914894 learned_addr learned my addr 172.21.15.122:0/588914894 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:19.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284f7fe640 1 -- 172.21.15.122:0/588914894 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 972156466 0 0) 0x7f28500a8fc0 con 0x7f28501498e0 2024-04-26T02:21:19.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284f7fe640 1 -- 172.21.15.122:0/588914894 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f283c003530 con 0x7f28501498e0 2024-04-26T02:21:19.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284f7fe640 1 -- 172.21.15.122:0/588914894 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 860443865 0 0) 0x7f283c003530 con 0x7f28501498e0 2024-04-26T02:21:19.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284f7fe640 1 -- 172.21.15.122:0/588914894 >> v1:172.21.15.87:6790/0 conn(0x7f28500724d0 legacy=0x7f2850143550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284f7fe640 1 -- 172.21.15.122:0/588914894 >> v1:172.21.15.87:6789/0 conn(0x7f285013cf90 legacy=0x7f285013d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284f7fe640 1 -- 172.21.15.122:0/588914894 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285013db10 con 0x7f28501498e0 2024-04-26T02:21:19.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284f7fe640 1 -- 172.21.15.122:0/588914894 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f28480019a0 con 0x7f28501498e0 2024-04-26T02:21:19.955 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284f7fe640 1 -- 172.21.15.122:0/588914894 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2848001c10 con 0x7f28501498e0 2024-04-26T02:21:19.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284f7fe640 1 -- 172.21.15.122:0/588914894 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f2848002020 con 0x7f28501498e0 2024-04-26T02:21:19.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f2856cd4640 1 -- 172.21.15.122:0/588914894 >> v1:172.21.15.122:6789/0 conn(0x7f28501498e0 legacy=0x7f285014bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f2856cd4640 1 -- 172.21.15.122:0/588914894 shutdown_connections 2024-04-26T02:21:19.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f2856cd4640 1 -- 172.21.15.122:0/588914894 >> 172.21.15.122:0/588914894 conn(0x7f28500b4a10 msgr2=0x7f28500b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:19.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f2856cd4640 1 -- 172.21.15.122:0/588914894 shutdown_connections 2024-04-26T02:21:19.956 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f2856cd4640 1 -- 172.21.15.122:0/588914894 wait complete. 2024-04-26T02:21:19.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f2856cd4640 1 Processor -- start 2024-04-26T02:21:19.958 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f2856cd4640 1 -- start start 2024-04-26T02:21:19.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f2856cd4640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2850140140 con 0x7f28500724d0 2024-04-26T02:21:19.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f2856cd4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2850143200 con 0x7f28501498e0 2024-04-26T02:21:19.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f2856cd4640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f285013ddd0 con 0x7f285013cf90 2024-04-26T02:21:19.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f2855271640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f28501498e0 0x7f28501dff40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45150/0 (socket says 172.21.15.122:45150) 2024-04-26T02:21:19.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f2855271640 1 -- 172.21.15.122:0/2012378128 learned_addr learned my addr 172.21.15.122:0/2012378128 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:19.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 587601274 0 0) 0x7f2850143200 con 0x7f28501498e0 2024-04-26T02:21:19.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2830003530 con 0x7f28501498e0 2024-04-26T02:21:19.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 990954957 0 0) 0x7f2830003530 con 0x7f28501498e0 2024-04-26T02:21:19.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2850143200 con 0x7f28501498e0 2024-04-26T02:21:19.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f2848002060 con 0x7f28501498e0 2024-04-26T02:21:19.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2785520935 0 0) 0x7f2850140140 con 0x7f28500724d0 2024-04-26T02:21:19.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2830003530 con 0x7f28500724d0 2024-04-26T02:21:19.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 228925192 0 0) 0x7f285013ddd0 con 0x7f285013cf90 2024-04-26T02:21:19.959 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2850140140 con 0x7f285013cf90 2024-04-26T02:21:19.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 <== mon.1 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2628562702 0 0) 0x7f2850143200 con 0x7f28501498e0 2024-04-26T02:21:19.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 >> v1:172.21.15.87:6790/0 conn(0x7f285013cf90 legacy=0x7f28501361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 >> v1:172.21.15.87:6789/0 conn(0x7f28500724d0 legacy=0x7f2850142870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:19.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285013dfa0 con 0x7f28501498e0 2024-04-26T02:21:19.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f2856cd4640 1 -- 172.21.15.122:0/2012378128 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f28501368d0 con 0x7f28501498e0 2024-04-26T02:21:19.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f2856cd4640 1 -- 172.21.15.122:0/2012378128 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2850136d90 con 0x7f28501498e0 2024-04-26T02:21:19.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 <== mon.1 v1:172.21.15.122:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2848005000 con 0x7f28501498e0 2024-04-26T02:21:19.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 <== mon.1 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f2848005230 con 0x7f28501498e0 2024-04-26T02:21:19.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 <== mon.1 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f28480057e0 con 0x7f28501498e0 2024-04-26T02:21:19.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.956+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 <== mon.1 v1:172.21.15.122:6789/0 8 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (unknown 4151114323 0 0) 0x7f2848006b40 con 0x7f28501498e0 2024-04-26T02:21:19.960 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.960+0000 7f2856cd4640 1 -- 172.21.15.122:0/2012378128 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2834005090 con 0x7f28501498e0 2024-04-26T02:21:19.961 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:19.960+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 <== mon.1 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2848005ad0 con 0x7f28501498e0 2024-04-26T02:21:20.027 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.028+0000 7f2856cd4640 1 -- 172.21.15.122:0/2012378128 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bfc148fb-9fd7-4a94-8287-52a4f38d7100", "id": 3} v 0) -- 0x7f2834005380 con 0x7f28501498e0 2024-04-26T02:21:20.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.032+0000 7f284d7fa640 1 -- 172.21.15.122:0/2012378128 <== mon.1 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bfc148fb-9fd7-4a94-8287-52a4f38d7100", "id": 3}]=0 v6) ==== 112+0+2 (unknown 398655299 0 525046306) 0x7f2848005dc0 con 0x7f28501498e0 2024-04-26T02:21:20.033 INFO:teuthology.orchestra.run.smithi122.stdout:3 2024-04-26T02:21:20.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.036+0000 7f2856cd4640 1 -- 172.21.15.122:0/2012378128 >> v1:172.21.15.122:6789/0 conn(0x7f28501498e0 legacy=0x7f28501dff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.036+0000 7f2856cd4640 1 -- 172.21.15.122:0/2012378128 shutdown_connections 2024-04-26T02:21:20.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.036+0000 7f2856cd4640 1 -- 172.21.15.122:0/2012378128 >> 172.21.15.122:0/2012378128 conn(0x7f28500b4a10 msgr2=0x7f285013bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:20.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.036+0000 7f2856cd4640 1 -- 172.21.15.122:0/2012378128 shutdown_connections 2024-04-26T02:21:20.034 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.036+0000 7f2856cd4640 1 -- 172.21.15.122:0/2012378128 wait complete. 2024-04-26T02:21:20.049 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new 3af86762-d77b-4c24-ac59-d0f20c4d423c 4 2024-04-26T02:21:20.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6c2112640 1 Processor -- start 2024-04-26T02:21:20.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6c2112640 1 -- start start 2024-04-26T02:21:20.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6c2112640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc0ae200 con 0x7fb6bc0acfb0 2024-04-26T02:21:20.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6c2112640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc0aec00 con 0x7fb6bc06b1b0 2024-04-26T02:21:20.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6c2112640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc072d20 con 0x7fb6bc06b590 2024-04-26T02:21:20.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6bb7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb6bc06b1b0 0x7fb6bc0b0790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45160/0 (socket says 172.21.15.122:45160) 2024-04-26T02:21:20.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6bb7fe640 1 -- 172.21.15.122:0/3886429004 learned_addr learned my addr 172.21.15.122:0/3886429004 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:20.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6ba7fc640 1 -- 172.21.15.122:0/3886429004 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3624952377 0 0) 0x7fb6bc0aec00 con 0x7fb6bc06b1b0 2024-04-26T02:21:20.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6ba7fc640 1 -- 172.21.15.122:0/3886429004 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6a8003530 con 0x7fb6bc06b1b0 2024-04-26T02:21:20.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6ba7fc640 1 -- 172.21.15.122:0/3886429004 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2504569357 0 0) 0x7fb6a8003530 con 0x7fb6bc06b1b0 2024-04-26T02:21:20.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6ba7fc640 1 -- 172.21.15.122:0/3886429004 >> v1:172.21.15.87:6790/0 conn(0x7fb6bc06b590 legacy=0x7fb6bc13f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6ba7fc640 1 -- 172.21.15.122:0/3886429004 >> v1:172.21.15.87:6789/0 conn(0x7fb6bc0acfb0 legacy=0x7fb6bc14d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.166 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6ba7fc640 1 -- 172.21.15.122:0/3886429004 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6bc072ef0 con 0x7fb6bc06b1b0 2024-04-26T02:21:20.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6ba7fc640 1 -- 172.21.15.122:0/3886429004 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb6ac001620 con 0x7fb6bc06b1b0 2024-04-26T02:21:20.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6ba7fc640 1 -- 172.21.15.122:0/3886429004 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6ac001880 con 0x7fb6bc06b1b0 2024-04-26T02:21:20.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.164+0000 7fb6ba7fc640 1 -- 172.21.15.122:0/3886429004 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb6ac001c50 con 0x7fb6bc06b1b0 2024-04-26T02:21:20.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6c2112640 1 -- 172.21.15.122:0/3886429004 >> v1:172.21.15.122:6789/0 conn(0x7fb6bc06b1b0 legacy=0x7fb6bc0b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6c2112640 1 -- 172.21.15.122:0/3886429004 shutdown_connections 2024-04-26T02:21:20.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6c2112640 1 -- 172.21.15.122:0/3886429004 >> 172.21.15.122:0/3886429004 conn(0x7fb6bc0b48a0 msgr2=0x7fb6bc0b4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:20.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6c2112640 1 -- 172.21.15.122:0/3886429004 shutdown_connections 2024-04-26T02:21:20.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6c2112640 1 -- 172.21.15.122:0/3886429004 wait complete. 2024-04-26T02:21:20.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6c2112640 1 Processor -- start 2024-04-26T02:21:20.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6c2112640 1 -- start start 2024-04-26T02:21:20.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6c2112640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc13b430 con 0x7fb6bc06b1b0 2024-04-26T02:21:20.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6c2112640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc13fee0 con 0x7fb6bc0acfb0 2024-04-26T02:21:20.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6c2112640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc1425f0 con 0x7fb6bc11cb90 2024-04-26T02:21:20.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6baffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb6bc0acfb0 0x7fb6bc1d75e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45174/0 (socket says 172.21.15.122:45174) 2024-04-26T02:21:20.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6baffd640 1 -- 172.21.15.122:0/3430541706 learned_addr learned my addr 172.21.15.122:0/3430541706 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:20.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3723056150 0 0) 0x7fb6bc13fee0 con 0x7fb6bc0acfb0 2024-04-26T02:21:20.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb69c003530 con 0x7fb6bc0acfb0 2024-04-26T02:21:20.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1016755916 0 0) 0x7fb69c003530 con 0x7fb6bc0acfb0 2024-04-26T02:21:20.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6bc13fee0 con 0x7fb6bc0acfb0 2024-04-26T02:21:20.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb6a4001980 con 0x7fb6bc0acfb0 2024-04-26T02:21:20.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4200451527 0 0) 0x7fb6bc1425f0 con 0x7fb6bc11cb90 2024-04-26T02:21:20.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb69c003530 con 0x7fb6bc11cb90 2024-04-26T02:21:20.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3612508709 0 0) 0x7fb6bc13b430 con 0x7fb6bc06b1b0 2024-04-26T02:21:20.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6bc1425f0 con 0x7fb6bc06b1b0 2024-04-26T02:21:20.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 <== mon.1 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2578112214 0 0) 0x7fb6bc13fee0 con 0x7fb6bc0acfb0 2024-04-26T02:21:20.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 >> v1:172.21.15.87:6790/0 conn(0x7fb6bc11cb90 legacy=0x7fb6bc13a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 >> v1:172.21.15.87:6789/0 conn(0x7fb6bc06b1b0 legacy=0x7fb6bc11c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6bc143040 con 0x7fb6bc0acfb0 2024-04-26T02:21:20.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6c2112640 1 -- 172.21.15.122:0/3430541706 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb6bc1d7cf0 con 0x7fb6bc0acfb0 2024-04-26T02:21:20.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6c2112640 1 -- 172.21.15.122:0/3430541706 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6bc13ae30 con 0x7fb6bc0acfb0 2024-04-26T02:21:20.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6baffd640 1 -- 172.21.15.122:0/3430541706 >> v1:172.21.15.122:6789/0 conn(0x7fb6bc0acfb0 legacy=0x7fb6bc1d75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T02:21:20.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6baffd640 1 -- 172.21.15.122:0/3430541706 >> v1:172.21.15.122:6789/0 conn(0x7fb6bc0acfb0 legacy=0x7fb6bc1d75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T02:21:20.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6baffd640 1 --1- 172.21.15.122:0/3430541706 >> v1:172.21.15.122:6789/0 conn(0x7fb6bc0acfb0 0x7fb6bc1d75e0 :-1 s=OPENED pgs=25 cs=1 l=1).handle_message read tag failed 2024-04-26T02:21:20.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6baffd640 1 --1- 172.21.15.122:0/3430541706 >> v1:172.21.15.122:6789/0 conn(0x7fb6bc0acfb0 0x7fb6bc1d75e0 :-1 s=OPENED pgs=25 cs=1 l=1).fault on lossy channel, failing 2024-04-26T02:21:20.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 >> v1:172.21.15.122:6789/0 conn(0x7fb6bc0acfb0 legacy=0x7fb6bc1d75e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T02:21:20.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb6bc13fee0 con 0x7fb6bc06b1b0 2024-04-26T02:21:20.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb6bc13b430 con 0x7fb69c004a40 2024-04-26T02:21:20.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb69c011f00 con 0x7fb6bc11cb90 2024-04-26T02:21:20.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 760611848 0 0) 0x7fb6bc13b430 con 0x7fb69c004a40 2024-04-26T02:21:20.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 --> v1:172.21.15.122:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fb69c010ed0 con 0x7fb69c004a40 2024-04-26T02:21:20.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.168+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 2963120567 0 0) 0x7fb69c010ed0 con 0x7fb69c004a40 2024-04-26T02:21:20.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.172+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 >> v1:172.21.15.87:6790/0 conn(0x7fb6bc11cb90 legacy=0x7fb69c00e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.173 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.172+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 >> v1:172.21.15.87:6789/0 conn(0x7fb6bc06b1b0 legacy=0x7fb69c00a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.172+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7fb69c013280 con 0x7fb69c004a40 2024-04-26T02:21:20.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.172+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb6ac001580 con 0x7fb69c004a40 2024-04-26T02:21:20.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.172+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6ac001820 con 0x7fb69c004a40 2024-04-26T02:21:20.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.172+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb6ac001a90 con 0x7fb69c004a40 2024-04-26T02:21:20.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.172+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 <== mon.1 v1:172.21.15.122:6789/0 6 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (unknown 743853035 0 0) 0x7fb6ac006190 con 0x7fb69c004a40 2024-04-26T02:21:20.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.172+0000 7fb6c2112640 1 -- 172.21.15.122:0/3430541706 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a0005090 con 0x7fb69c004a40 2024-04-26T02:21:20.174 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.172+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 <== mon.1 v1:172.21.15.122:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb6ac005d50 con 0x7fb69c004a40 2024-04-26T02:21:20.194 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:20.196+0000 7fc960988180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T02:21:20.198 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:20.201+0000 7f024dae3180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T02:21:20.244 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.244+0000 7fb6c2112640 1 -- 172.21.15.122:0/3430541706 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3af86762-d77b-4c24-ac59-d0f20c4d423c", "id": 4} v 0) -- 0x7fb6a0005380 con 0x7fb69c004a40 2024-04-26T02:21:20.248 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.248+0000 7fb6b3fff640 1 -- 172.21.15.122:0/3430541706 <== mon.1 v1:172.21.15.122:6789/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3af86762-d77b-4c24-ac59-d0f20c4d423c", "id": 4}]=0 v7) ==== 112+0+2 (unknown 61620716 0 1697012583) 0x7fb6ac006430 con 0x7fb69c004a40 2024-04-26T02:21:20.249 INFO:teuthology.orchestra.run.smithi122.stdout:4 2024-04-26T02:21:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.248+0000 7fb6c2112640 1 -- 172.21.15.122:0/3430541706 >> v1:172.21.15.122:6789/0 conn(0x7fb69c004a40 legacy=0x7fb69c004e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.248+0000 7fb6c2112640 1 -- 172.21.15.122:0/3430541706 shutdown_connections 2024-04-26T02:21:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.248+0000 7fb6c2112640 1 -- 172.21.15.122:0/3430541706 >> 172.21.15.122:0/3430541706 conn(0x7fb6bc0b48a0 msgr2=0x7fb6bc0af9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.248+0000 7fb6c2112640 1 -- 172.21.15.122:0/3430541706 shutdown_connections 2024-04-26T02:21:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.248+0000 7fb6c2112640 1 -- 172.21.15.122:0/3430541706 wait complete. 2024-04-26T02:21:20.251 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:20.253+0000 7f024dae3180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T02:21:20.252 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:20.252+0000 7fc960988180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T02:21:20.264 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new 6a28bfec-7754-4a56-bbd1-0d1d2655fde7 5 2024-04-26T02:21:20.304 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:20.305+0000 7f024dae3180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T02:21:20.310 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:20.312+0000 7fc960988180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T02:21:20.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.368+0000 7f5249768640 1 Processor -- start 2024-04-26T02:21:20.369 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.368+0000 7f5249768640 1 -- start start 2024-04-26T02:21:20.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5249768640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f524413f280 con 0x7f52440707c0 2024-04-26T02:21:20.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5249768640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5244072620 con 0x7f52441389e0 2024-04-26T02:21:20.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5249768640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5244072900 con 0x7f52440703e0 2024-04-26T02:21:20.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f52437fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f52441389e0 0x7f524414a6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45196/0 (socket says 172.21.15.122:45196) 2024-04-26T02:21:20.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f52437fe640 1 -- 172.21.15.122:0/3156178438 learned_addr learned my addr 172.21.15.122:0/3156178438 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:20.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5241ffb640 1 -- 172.21.15.122:0/3156178438 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 628751416 0 0) 0x7f5244072620 con 0x7f52441389e0 2024-04-26T02:21:20.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5241ffb640 1 -- 172.21.15.122:0/3156178438 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5230003530 con 0x7f52441389e0 2024-04-26T02:21:20.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5241ffb640 1 -- 172.21.15.122:0/3156178438 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2942757673 0 0) 0x7f5230003530 con 0x7f52441389e0 2024-04-26T02:21:20.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5241ffb640 1 -- 172.21.15.122:0/3156178438 >> v1:172.21.15.87:6790/0 conn(0x7f52440703e0 legacy=0x7f52440af410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5241ffb640 1 -- 172.21.15.122:0/3156178438 >> v1:172.21.15.87:6789/0 conn(0x7f52440707c0 legacy=0x7f524413f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5241ffb640 1 -- 172.21.15.122:0/3156178438 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52440715a0 con 0x7f52441389e0 2024-04-26T02:21:20.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5241ffb640 1 -- 172.21.15.122:0/3156178438 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f523c0019a0 con 0x7f52441389e0 2024-04-26T02:21:20.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5241ffb640 1 -- 172.21.15.122:0/3156178438 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f523c001c90 con 0x7f52441389e0 2024-04-26T02:21:20.371 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5241ffb640 1 -- 172.21.15.122:0/3156178438 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f523c0020a0 con 0x7f52441389e0 2024-04-26T02:21:20.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5249768640 1 -- 172.21.15.122:0/3156178438 >> v1:172.21.15.122:6789/0 conn(0x7f52441389e0 legacy=0x7f524414a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5249768640 1 -- 172.21.15.122:0/3156178438 shutdown_connections 2024-04-26T02:21:20.372 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5249768640 1 -- 172.21.15.122:0/3156178438 >> 172.21.15.122:0/3156178438 conn(0x7f52440b4a10 msgr2=0x7f52440b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:20.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5249768640 1 -- 172.21.15.122:0/3156178438 shutdown_connections 2024-04-26T02:21:20.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5249768640 1 -- 172.21.15.122:0/3156178438 wait complete. 2024-04-26T02:21:20.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5249768640 1 Processor -- start 2024-04-26T02:21:20.373 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5249768640 1 -- start start 2024-04-26T02:21:20.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5249768640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f524412fb40 con 0x7f52441389e0 2024-04-26T02:21:20.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5249768640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52441334f0 con 0x7f52440703e0 2024-04-26T02:21:20.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5249768640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5244130020 con 0x7f52440707c0 2024-04-26T02:21:20.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5242ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f52440703e0 0x7f52441d72d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45202/0 (socket says 172.21.15.122:45202) 2024-04-26T02:21:20.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f5242ffd640 1 -- 172.21.15.122:0/1036682356 learned_addr learned my addr 172.21.15.122:0/1036682356 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:20.375 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1901000409 0 0) 0x7f52441334f0 con 0x7f52440703e0 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5224003530 con 0x7f52440703e0 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3416844522 0 0) 0x7f5224003530 con 0x7f52440703e0 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52441334f0 con 0x7f52440703e0 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f522c001960 con 0x7f52440703e0 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3087564661 0 0) 0x7f5244130020 con 0x7f52440707c0 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5224003530 con 0x7f52440707c0 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2830228245 0 0) 0x7f524412fb40 con 0x7f52441389e0 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.372+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5244130020 con 0x7f52441389e0 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.376+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 <== mon.1 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2019046533 0 0) 0x7f52441334f0 con 0x7f52440703e0 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.376+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 >> v1:172.21.15.87:6790/0 conn(0x7f52440707c0 legacy=0x7f524412ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.376+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 >> v1:172.21.15.87:6789/0 conn(0x7f52441389e0 legacy=0x7f52441e2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.376+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52441308b0 con 0x7f52440703e0 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.376+0000 7f5249768640 1 -- 172.21.15.122:0/1036682356 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f52441d79e0 con 0x7f52440703e0 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.376+0000 7f5249768640 1 -- 172.21.15.122:0/1036682356 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f52441305a0 con 0x7f52440703e0 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.376+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 <== mon.1 v1:172.21.15.122:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f522c001ca0 con 0x7f52440703e0 2024-04-26T02:21:20.376 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.376+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 <== mon.1 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f522c003af0 con 0x7f52440703e0 2024-04-26T02:21:20.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.376+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 <== mon.1 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f522c0040a0 con 0x7f52440703e0 2024-04-26T02:21:20.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.376+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 <== mon.1 v1:172.21.15.122:6789/0 8 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (unknown 3465325394 0 0) 0x7f522c006ab0 con 0x7f52440703e0 2024-04-26T02:21:20.377 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.376+0000 7f5249768640 1 -- 172.21.15.122:0/1036682356 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5228005090 con 0x7f52440703e0 2024-04-26T02:21:20.380 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:20.381+0000 7f024dae3180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T02:21:20.390 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.392+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 <== mon.1 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f522c006640 con 0x7f52440703e0 2024-04-26T02:21:20.392 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:20.392+0000 7fc960988180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T02:21:20.432 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:20.433+0000 7f024dae3180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T02:21:20.444 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:20.444+0000 7fc960988180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T02:21:20.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.452+0000 7f5249768640 1 -- 172.21.15.122:0/1036682356 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6a28bfec-7754-4a56-bbd1-0d1d2655fde7", "id": 5} v 0) -- 0x7f5228005380 con 0x7f52440703e0 2024-04-26T02:21:20.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.460+0000 7f523b7fe640 1 -- 172.21.15.122:0/1036682356 <== mon.1 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6a28bfec-7754-4a56-bbd1-0d1d2655fde7", "id": 5}]=0 v8) ==== 112+0+2 (unknown 2849864621 0 1988412176) 0x7f522c0057d0 con 0x7f52440703e0 2024-04-26T02:21:20.460 INFO:teuthology.orchestra.run.smithi122.stdout:5 2024-04-26T02:21:20.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.460+0000 7f5249768640 1 -- 172.21.15.122:0/1036682356 >> v1:172.21.15.122:6789/0 conn(0x7f52440703e0 legacy=0x7f52441d72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.460+0000 7f5249768640 1 -- 172.21.15.122:0/1036682356 shutdown_connections 2024-04-26T02:21:20.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.460+0000 7f5249768640 1 -- 172.21.15.122:0/1036682356 >> 172.21.15.122:0/1036682356 conn(0x7f52440b4a10 msgr2=0x7f52440b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:20.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.460+0000 7f5249768640 1 -- 172.21.15.122:0/1036682356 shutdown_connections 2024-04-26T02:21:20.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.460+0000 7f5249768640 1 -- 172.21.15.122:0/1036682356 wait complete. 2024-04-26T02:21:20.475 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new db087c2e-bd80-4863-a1db-5fbd8f9fe083 6 2024-04-26T02:21:20.508 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:20.509+0000 7f024dae3180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T02:21:20.524 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:20.524+0000 7fc960988180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T02:21:20.561 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:20.561+0000 7f024dae3180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T02:21:20.578 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:20.580+0000 7fc960988180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T02:21:20.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.584+0000 7ff0fbde5640 1 Processor -- start 2024-04-26T02:21:20.584 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.584+0000 7ff0fbde5640 1 -- start start 2024-04-26T02:21:20.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.584+0000 7ff0fbde5640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f40a8cc0 con 0x7ff0f4066e50 2024-04-26T02:21:20.586 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.584+0000 7ff0fbde5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f40a8f30 con 0x7ff0f4072b90 2024-04-26T02:21:20.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.584+0000 7ff0fbde5640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f4072070 con 0x7ff0f4067230 2024-04-26T02:21:20.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.584+0000 7ff0fa382640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff0f4072b90 0x7ff0f4143ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45212/0 (socket says 172.21.15.122:45212) 2024-04-26T02:21:20.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.584+0000 7ff0fa382640 1 -- 172.21.15.122:0/4260770666 learned_addr learned my addr 172.21.15.122:0/4260770666 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:20.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.584+0000 7ff0f8b7f640 1 -- 172.21.15.122:0/4260770666 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4044848555 0 0) 0x7ff0e40014a0 con 0x7ff0f4072b90 2024-04-26T02:21:20.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.584+0000 7ff0f8b7f640 1 -- 172.21.15.122:0/4260770666 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0dc003530 con 0x7ff0f4072b90 2024-04-26T02:21:20.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.584+0000 7ff0f8b7f640 1 -- 172.21.15.122:0/4260770666 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 513375733 0 0) 0x7ff0dc003530 con 0x7ff0f4072b90 2024-04-26T02:21:20.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.584+0000 7ff0f8b7f640 1 -- 172.21.15.122:0/4260770666 >> v1:172.21.15.87:6790/0 conn(0x7ff0f4067230 legacy=0x7ff0f4143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.584+0000 7ff0f8b7f640 1 -- 172.21.15.122:0/4260770666 >> v1:172.21.15.87:6789/0 conn(0x7ff0f4066e50 legacy=0x7ff0f40adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.584+0000 7ff0f8b7f640 1 -- 172.21.15.122:0/4260770666 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f414f870 con 0x7ff0f4072b90 2024-04-26T02:21:20.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.584+0000 7ff0f8b7f640 1 -- 172.21.15.122:0/4260770666 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7ff0e4001b70 con 0x7ff0f4072b90 2024-04-26T02:21:20.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f8b7f640 1 -- 172.21.15.122:0/4260770666 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff0e4001e60 con 0x7ff0f4072b90 2024-04-26T02:21:20.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f8b7f640 1 -- 172.21.15.122:0/4260770666 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7ff0e4004c30 con 0x7ff0f4072b90 2024-04-26T02:21:20.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0fbde5640 1 -- 172.21.15.122:0/4260770666 >> v1:172.21.15.122:6789/0 conn(0x7ff0f4072b90 legacy=0x7ff0f4143ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0fbde5640 1 -- 172.21.15.122:0/4260770666 shutdown_connections 2024-04-26T02:21:20.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0fbde5640 1 -- 172.21.15.122:0/4260770666 >> 172.21.15.122:0/4260770666 conn(0x7ff0f40b44b0 msgr2=0x7ff0f40b48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:20.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0fbde5640 1 -- 172.21.15.122:0/4260770666 shutdown_connections 2024-04-26T02:21:20.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0fbde5640 1 -- 172.21.15.122:0/4260770666 wait complete. 2024-04-26T02:21:20.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0fbde5640 1 Processor -- start 2024-04-26T02:21:20.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0fbde5640 1 -- start start 2024-04-26T02:21:20.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0fbde5640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f413e3e0 con 0x7ff0f4072b90 2024-04-26T02:21:20.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0fbde5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f412bb10 con 0x7ff0f4066e50 2024-04-26T02:21:20.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0fbde5640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f412bce0 con 0x7ff0f41426e0 2024-04-26T02:21:20.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f9b81640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff0f4066e50 0x7ff0f412b400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45220/0 (socket says 172.21.15.122:45220) 2024-04-26T02:21:20.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f9b81640 1 -- 172.21.15.122:0/801712896 learned_addr learned my addr 172.21.15.122:0/801712896 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:20.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1268827091 0 0) 0x7ff0f412bb10 con 0x7ff0f4066e50 2024-04-26T02:21:20.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0e0003530 con 0x7ff0f4066e50 2024-04-26T02:21:20.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2387469770 0 0) 0x7ff0e0003530 con 0x7ff0f4066e50 2024-04-26T02:21:20.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0f412bb10 con 0x7ff0f4066e50 2024-04-26T02:21:20.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7ff0ec001920 con 0x7ff0f4066e50 2024-04-26T02:21:20.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2159686620 0 0) 0x7ff0f412bce0 con 0x7ff0f41426e0 2024-04-26T02:21:20.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0e0003530 con 0x7ff0f41426e0 2024-04-26T02:21:20.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1033685380 0 0) 0x7ff0f413e3e0 con 0x7ff0f4072b90 2024-04-26T02:21:20.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0f412bce0 con 0x7ff0f4072b90 2024-04-26T02:21:20.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 <== mon.1 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3694113316 0 0) 0x7ff0f412bb10 con 0x7ff0f4066e50 2024-04-26T02:21:20.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 >> v1:172.21.15.87:6790/0 conn(0x7ff0f41426e0 legacy=0x7ff0f41d9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 >> v1:172.21.15.87:6789/0 conn(0x7ff0f4072b90 legacy=0x7ff0f41d8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f412ce80 con 0x7ff0f4066e50 2024-04-26T02:21:20.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0fbde5640 1 -- 172.21.15.122:0/801712896 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff0f412d170 con 0x7ff0f4066e50 2024-04-26T02:21:20.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0fbde5640 1 -- 172.21.15.122:0/801712896 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff0f413dee0 con 0x7ff0f4066e50 2024-04-26T02:21:20.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 <== mon.1 v1:172.21.15.122:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff0ec001d30 con 0x7ff0f4066e50 2024-04-26T02:21:20.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 <== mon.1 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7ff0ec003af0 con 0x7ff0f4066e50 2024-04-26T02:21:20.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 <== mon.1 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff0ec004020 con 0x7ff0f4066e50 2024-04-26T02:21:20.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 <== mon.1 v1:172.21.15.122:6789/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 3672326418 0 0) 0x7ff0ec006b20 con 0x7ff0f4066e50 2024-04-26T02:21:20.593 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.588+0000 7ff0fbde5640 1 -- 172.21.15.122:0/801712896 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0d4005090 con 0x7ff0f4066e50 2024-04-26T02:21:20.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.592+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 <== mon.1 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff0ec006530 con 0x7ff0f4066e50 2024-04-26T02:21:20.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.656+0000 7ff0fbde5640 1 -- 172.21.15.122:0/801712896 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd new", "uuid": "db087c2e-bd80-4863-a1db-5fbd8f9fe083", "id": 6} v 0) -- 0x7ff0d4005380 con 0x7ff0f4066e50 2024-04-26T02:21:20.662 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.664+0000 7ff0f27fc640 1 -- 172.21.15.122:0/801712896 <== mon.1 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "db087c2e-bd80-4863-a1db-5fbd8f9fe083", "id": 6}]=0 v9) ==== 112+0+2 (unknown 3331595748 0 1113811849) 0x7ff0ec0067e0 con 0x7ff0f4066e50 2024-04-26T02:21:20.662 INFO:teuthology.orchestra.run.smithi122.stdout:6 2024-04-26T02:21:20.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.664+0000 7ff0fbde5640 1 -- 172.21.15.122:0/801712896 >> v1:172.21.15.122:6789/0 conn(0x7ff0f4066e50 legacy=0x7ff0f412b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.664+0000 7ff0fbde5640 1 -- 172.21.15.122:0/801712896 shutdown_connections 2024-04-26T02:21:20.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.664+0000 7ff0fbde5640 1 -- 172.21.15.122:0/801712896 >> 172.21.15.122:0/801712896 conn(0x7ff0f40b44b0 msgr2=0x7ff0f40aecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:20.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.664+0000 7ff0fbde5640 1 -- 172.21.15.122:0/801712896 shutdown_connections 2024-04-26T02:21:20.663 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.664+0000 7ff0fbde5640 1 -- 172.21.15.122:0/801712896 wait complete. 2024-04-26T02:21:20.677 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new ccf8f7a9-c44f-44f4-91d1-9f197803f67d 7 2024-04-26T02:21:20.715 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:20.713+0000 7f024dae3180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T02:21:20.734 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:20.736+0000 7fc960988180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T02:21:20.774 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:20.777+0000 7f024dae3180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T02:21:20.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1e7430640 1 Processor -- start 2024-04-26T02:21:20.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1e7430640 1 -- start start 2024-04-26T02:21:20.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1e7430640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e00723c0 con 0x7fd1e0070540 2024-04-26T02:21:20.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1e7430640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e00726a0 con 0x7fd1e0070160 2024-04-26T02:21:20.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1e7430640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e0070fa0 con 0x7fd1e00ad520 2024-04-26T02:21:20.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1e51cc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd1e0070160 0x7fd1e0073210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45224/0 (socket says 172.21.15.122:45224) 2024-04-26T02:21:20.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1e51cc640 1 -- 172.21.15.122:0/2450795849 learned_addr learned my addr 172.21.15.122:0/2450795849 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:20.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1d3fff640 1 -- 172.21.15.122:0/2450795849 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4011865018 0 0) 0x7fd1e00726a0 con 0x7fd1e0070160 2024-04-26T02:21:20.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1d3fff640 1 -- 172.21.15.122:0/2450795849 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1cc003450 con 0x7fd1e0070160 2024-04-26T02:21:20.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1d3fff640 1 -- 172.21.15.122:0/2450795849 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3924832093 0 0) 0x7fd1cc003450 con 0x7fd1e0070160 2024-04-26T02:21:20.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1d3fff640 1 -- 172.21.15.122:0/2450795849 >> v1:172.21.15.87:6790/0 conn(0x7fd1e00ad520 legacy=0x7fd1e0149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1d3fff640 1 -- 172.21.15.122:0/2450795849 >> v1:172.21.15.87:6789/0 conn(0x7fd1e0070540 legacy=0x7fd1e0141a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1d3fff640 1 -- 172.21.15.122:0/2450795849 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e0072cf0 con 0x7fd1e0070160 2024-04-26T02:21:20.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1d3fff640 1 -- 172.21.15.122:0/2450795849 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fd1dc001620 con 0x7fd1e0070160 2024-04-26T02:21:20.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1d3fff640 1 -- 172.21.15.122:0/2450795849 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd1dc001910 con 0x7fd1e0070160 2024-04-26T02:21:20.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.784+0000 7fd1d3fff640 1 -- 172.21.15.122:0/2450795849 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fd1dc001ce0 con 0x7fd1e0070160 2024-04-26T02:21:20.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1e7430640 1 -- 172.21.15.122:0/2450795849 >> v1:172.21.15.122:6789/0 conn(0x7fd1e0070160 legacy=0x7fd1e0073210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1e7430640 1 -- 172.21.15.122:0/2450795849 shutdown_connections 2024-04-26T02:21:20.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1e7430640 1 -- 172.21.15.122:0/2450795849 >> 172.21.15.122:0/2450795849 conn(0x7fd1e00b4180 msgr2=0x7fd1e00b4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:20.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1e7430640 1 -- 172.21.15.122:0/2450795849 shutdown_connections 2024-04-26T02:21:20.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1e7430640 1 -- 172.21.15.122:0/2450795849 wait complete. 2024-04-26T02:21:20.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1e7430640 1 Processor -- start 2024-04-26T02:21:20.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1e7430640 1 -- start start 2024-04-26T02:21:20.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1e7430640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e01d8110 con 0x7fd1e00ad520 2024-04-26T02:21:20.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1e7430640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e01d74e0 con 0x7fd1e0070540 2024-04-26T02:21:20.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1e7430640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e01cd880 con 0x7fd1e0070160 2024-04-26T02:21:20.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1e49cb640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd1e0070540 0x7fd1e01d7a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45236/0 (socket says 172.21.15.122:45236) 2024-04-26T02:21:20.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1e49cb640 1 -- 172.21.15.122:0/3927012614 learned_addr learned my addr 172.21.15.122:0/3927012614 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-04-26T02:21:20.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 418972583 0 0) 0x7fd1e01d74e0 con 0x7fd1e0070540 2024-04-26T02:21:20.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1c4003530 con 0x7fd1e0070540 2024-04-26T02:21:20.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 696186886 0 0) 0x7fd1c4003530 con 0x7fd1e0070540 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1e01d74e0 con 0x7fd1e0070540 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fd1d4001980 con 0x7fd1e0070540 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 142411110 0 0) 0x7fd1e01cd880 con 0x7fd1e0070160 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1c4003530 con 0x7fd1e0070160 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2901987857 0 0) 0x7fd1e01d8110 con 0x7fd1e00ad520 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1e01cd880 con 0x7fd1e00ad520 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 <== mon.1 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 712027762 0 0) 0x7fd1e01d74e0 con 0x7fd1e0070540 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 >> v1:172.21.15.87:6790/0 conn(0x7fd1e0070160 legacy=0x7fd1e01caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 >> v1:172.21.15.87:6789/0 conn(0x7fd1e00ad520 legacy=0x7fd1e01e1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e01ca570 con 0x7fd1e0070540 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1e7430640 1 -- 172.21.15.122:0/3927012614 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e01c9e70 con 0x7fd1e0070540 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 <== mon.1 v1:172.21.15.122:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd1d4003950 con 0x7fd1e0070540 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1e7430640 1 -- 172.21.15.122:0/3927012614 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd1e01e31e0 con 0x7fd1e0070540 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 <== mon.1 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fd1d4003d20 con 0x7fd1e0070540 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 <== mon.1 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd1d4004250 con 0x7fd1e0070540 2024-04-26T02:21:20.791 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.788+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 <== mon.1 v1:172.21.15.122:6789/0 8 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (unknown 3702180446 0 0) 0x7fd1d4006c80 con 0x7fd1e0070540 2024-04-26T02:21:20.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.792+0000 7fd1e7430640 1 -- 172.21.15.122:0/3927012614 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1c0005090 con 0x7fd1e0070540 2024-04-26T02:21:20.792 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.792+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 <== mon.1 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd1d4006620 con 0x7fd1e0070540 2024-04-26T02:21:20.798 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:20.800+0000 7fc960988180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T02:21:20.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.856+0000 7fd1e7430640 1 -- 172.21.15.122:0/3927012614 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ccf8f7a9-c44f-44f4-91d1-9f197803f67d", "id": 7} v 0) -- 0x7fd1c0005380 con 0x7fd1e0070540 2024-04-26T02:21:20.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.860+0000 7fd1d1ffb640 1 -- 172.21.15.122:0/3927012614 <== mon.1 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ccf8f7a9-c44f-44f4-91d1-9f197803f67d", "id": 7}]=0 v10) ==== 112+0+2 (unknown 2070908896 0 1371665406) 0x7fd1d40068d0 con 0x7fd1e0070540 2024-04-26T02:21:20.859 INFO:teuthology.orchestra.run.smithi122.stdout:7 2024-04-26T02:21:20.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.860+0000 7fd1e7430640 1 -- 172.21.15.122:0/3927012614 >> v1:172.21.15.122:6789/0 conn(0x7fd1e0070540 legacy=0x7fd1e01d7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:20.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.860+0000 7fd1e7430640 1 -- 172.21.15.122:0/3927012614 shutdown_connections 2024-04-26T02:21:20.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.860+0000 7fd1e7430640 1 -- 172.21.15.122:0/3927012614 >> 172.21.15.122:0/3927012614 conn(0x7fd1e00b4180 msgr2=0x7fd1e00ac8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:20.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.860+0000 7fd1e7430640 1 -- 172.21.15.122:0/3927012614 shutdown_connections 2024-04-26T02:21:20.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-26T02:21:20.860+0000 7fd1e7430640 1 -- 172.21.15.122:0/3927012614 wait complete. 2024-04-26T02:21:20.875 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-26T02:21:20.875 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-26T02:21:20.877 INFO:tasks.ceph.osd.0:Started 2024-04-26T02:21:20.878 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-26T02:21:20.878 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-26T02:21:20.880 INFO:tasks.ceph.osd.1:Started 2024-04-26T02:21:20.880 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-26T02:21:20.880 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-26T02:21:20.883 INFO:tasks.ceph.osd.2:Started 2024-04-26T02:21:20.883 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-26T02:21:20.883 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-26T02:21:20.885 INFO:tasks.ceph.osd.3:Started 2024-04-26T02:21:20.886 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-26T02:21:20.886 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-26T02:21:20.888 INFO:tasks.ceph.osd.4:Started 2024-04-26T02:21:20.888 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-26T02:21:20.889 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-26T02:21:20.890 INFO:tasks.ceph.osd.5:Started 2024-04-26T02:21:20.890 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-26T02:21:20.890 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-26T02:21:20.893 INFO:tasks.ceph.osd.6:Started 2024-04-26T02:21:20.893 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-26T02:21:20.893 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-26T02:21:20.895 INFO:tasks.ceph.osd.7:Started 2024-04-26T02:21:20.895 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T02:21:21.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc3e02a640 1 Processor -- start 2024-04-26T02:21:21.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc3e02a640 1 -- start start 2024-04-26T02:21:21.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc3e02a640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc380a9f60 con 0x7fbc3813faf0 2024-04-26T02:21:21.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc3e02a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc380aadd0 con 0x7fbc380a8b70 2024-04-26T02:21:21.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc3e02a640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc380ab0b0 con 0x7fbc380b23f0 2024-04-26T02:21:21.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc37fff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fbc3813faf0 0x7fbc3814da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:35418/0 (socket says 172.21.15.87:35418) 2024-04-26T02:21:21.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc2effd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fbc380b23f0 0x7fbc3813f3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:55700/0 (socket says 172.21.15.87:55700) 2024-04-26T02:21:21.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc37fff640 1 -- 172.21.15.87:0/2438307276 learned_addr learned my addr 172.21.15.87:0/2438307276 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:21.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2914852564 0 0) 0x7fbc380ab0b0 con 0x7fbc380b23f0 2024-04-26T02:21:21.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc24003530 con 0x7fbc380b23f0 2024-04-26T02:21:21.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 165819355 0 0) 0x7fbc380a9f60 con 0x7fbc3813faf0 2024-04-26T02:21:21.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc380ab0b0 con 0x7fbc3813faf0 2024-04-26T02:21:21.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3922541903 0 0) 0x7fbc380aadd0 con 0x7fbc380a8b70 2024-04-26T02:21:21.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc380a9f60 con 0x7fbc380a8b70 2024-04-26T02:21:21.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4135883584 0 0) 0x7fbc380ab0b0 con 0x7fbc3813faf0 2024-04-26T02:21:21.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 >> v1:172.21.15.87:6790/0 conn(0x7fbc380b23f0 legacy=0x7fbc3813f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 >> v1:172.21.15.122:6789/0 conn(0x7fbc380a8b70 legacy=0x7fbc381427e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc381422c0 con 0x7fbc3813faf0 2024-04-26T02:21:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fbc30001980 con 0x7fbc3813faf0 2024-04-26T02:21:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc37fff640 1 -- 172.21.15.87:0/2438307276 >> v1:172.21.15.87:6789/0 conn(0x7fbc3813faf0 legacy=0x7fbc3814da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-04-26T02:21:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc37fff640 1 -- 172.21.15.87:0/2438307276 >> v1:172.21.15.87:6789/0 conn(0x7fbc3813faf0 legacy=0x7fbc3814da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T02:21:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc37fff640 1 --1- 172.21.15.87:0/2438307276 >> v1:172.21.15.87:6789/0 conn(0x7fbc3813faf0 0x7fbc3814da50 :-1 s=OPENED pgs=44 cs=1 l=1).handle_message read tag failed 2024-04-26T02:21:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc37fff640 1 --1- 172.21.15.87:0/2438307276 >> v1:172.21.15.87:6789/0 conn(0x7fbc3813faf0 0x7fbc3814da50 :-1 s=OPENED pgs=44 cs=1 l=1).fault on lossy channel, failing 2024-04-26T02:21:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 >> v1:172.21.15.87:6789/0 conn(0x7fbc3813faf0 legacy=0x7fbc3814da50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T02:21:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbc380ab0b0 con 0x7fbc24004e70 2024-04-26T02:21:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbc380aadd0 con 0x7fbc380a8b70 2024-04-26T02:21:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbc24005490 con 0x7fbc380b23f0 2024-04-26T02:21:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3966721780 0 0) 0x7fbc380ab0b0 con 0x7fbc24004e70 2024-04-26T02:21:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 --> v1:172.21.15.87:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fbc2400f110 con 0x7fbc24004e70 2024-04-26T02:21:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2929212216 0 0) 0x7fbc24005490 con 0x7fbc380b23f0 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.017+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 --> v1:172.21.15.87:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fbc380ab0b0 con 0x7fbc380b23f0 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2536015417 0 0) 0x7fbc380aadd0 con 0x7fbc380a8b70 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 --> v1:172.21.15.122:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fbc24005490 con 0x7fbc380a8b70 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 4195369310 0 0) 0x7fbc2400f110 con 0x7fbc24004e70 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 >> v1:172.21.15.87:6790/0 conn(0x7fbc380b23f0 legacy=0x7fbc2400d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 >> v1:172.21.15.122:6789/0 conn(0x7fbc380a8b70 legacy=0x7fbc24003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=2+}) -- 0x7fbc24012140 con 0x7fbc24004e70 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fbc28001740 con 0x7fbc24004e70 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc36ffd640 1 -- 172.21.15.87:0/2438307276 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc28001960 con 0x7fbc24004e70 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc3e02a640 1 -- 172.21.15.87:0/2438307276 >> v1:172.21.15.87:6789/0 conn(0x7fbc24004e70 legacy=0x7fbc240037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc3e02a640 1 -- 172.21.15.87:0/2438307276 shutdown_connections 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc3e02a640 1 -- 172.21.15.87:0/2438307276 >> 172.21.15.87:0/2438307276 conn(0x7fbc380b7a70 msgr2=0x7fbc380b7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc3e02a640 1 -- 172.21.15.87:0/2438307276 shutdown_connections 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc3e02a640 1 -- 172.21.15.87:0/2438307276 wait complete. 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc3e02a640 1 Processor -- start 2024-04-26T02:21:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc3e02a640 1 -- start start 2024-04-26T02:21:21.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc3e02a640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc381387e0 con 0x7fbc380b23f0 2024-04-26T02:21:21.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc3e02a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc38138b40 con 0x7fbc380a8b70 2024-04-26T02:21:21.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc3e02a640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc38138d10 con 0x7fbc24004e70 2024-04-26T02:21:21.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc377fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fbc24004e70 0x7fbc381da340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:55704/0 (socket says 172.21.15.87:55704) 2024-04-26T02:21:21.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc377fe640 1 -- 172.21.15.87:0/2821792017 learned_addr learned my addr 172.21.15.87:0/2821792017 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:21.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4218441888 0 0) 0x7fbc38138d10 con 0x7fbc24004e70 2024-04-26T02:21:21.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc18003530 con 0x7fbc24004e70 2024-04-26T02:21:21.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1159370662 0 0) 0x7fbc381387e0 con 0x7fbc380b23f0 2024-04-26T02:21:21.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc38138d10 con 0x7fbc380b23f0 2024-04-26T02:21:21.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1916155750 0 0) 0x7fbc18003530 con 0x7fbc24004e70 2024-04-26T02:21:21.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc381387e0 con 0x7fbc24004e70 2024-04-26T02:21:21.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fbc28001ca0 con 0x7fbc24004e70 2024-04-26T02:21:21.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 397545401 0 0) 0x7fbc38138d10 con 0x7fbc380b23f0 2024-04-26T02:21:21.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc18003530 con 0x7fbc380b23f0 2024-04-26T02:21:21.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1056857744 0 0) 0x7fbc38138b40 con 0x7fbc380a8b70 2024-04-26T02:21:21.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc38138d10 con 0x7fbc380a8b70 2024-04-26T02:21:21.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fbc30001740 con 0x7fbc380b23f0 2024-04-26T02:21:21.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2473506428 0 0) 0x7fbc381387e0 con 0x7fbc24004e70 2024-04-26T02:21:21.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 >> v1:172.21.15.122:6789/0 conn(0x7fbc380a8b70 legacy=0x7fbc381379c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:21.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 >> v1:172.21.15.87:6789/0 conn(0x7fbc380b23f0 legacy=0x7fbc381380d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:21.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc3813e300 con 0x7fbc24004e70 2024-04-26T02:21:21.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc3e02a640 1 -- 172.21.15.87:0/2821792017 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbc3813e590 con 0x7fbc24004e70 2024-04-26T02:21:21.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc3e02a640 1 -- 172.21.15.87:0/2821792017 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbc38146170 con 0x7fbc24004e70 2024-04-26T02:21:21.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc28001960 con 0x7fbc24004e70 2024-04-26T02:21:21.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fbc28003af0 con 0x7fbc24004e70 2024-04-26T02:21:21.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbc280040e0 con 0x7fbc24004e70 2024-04-26T02:21:21.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.021+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (unknown 16898136 0 0) 0x7fbc28006540 con 0x7fbc24004e70 2024-04-26T02:21:21.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.025+0000 7fbc3e02a640 1 -- 172.21.15.87:0/2821792017 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc1c005090 con 0x7fbc24004e70 2024-04-26T02:21:21.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.025+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbc28006820 con 0x7fbc24004e70 2024-04-26T02:21:21.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.117+0000 7fbc3e02a640 1 -- 172.21.15.87:0/2821792017 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc1c005380 con 0x7fbc24004e70 2024-04-26T02:21:21.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.121+0000 7fbc34ff9640 1 -- 172.21.15.87:0/2821792017 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (unknown 4040575075 0 37819316) 0x7fbc28006a80 con 0x7fbc24004e70 2024-04-26T02:21:21.122 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:21:21.122 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":10,"fsid":"6232cbab-48ed-4660-9de8-286f4c0e9a32","created":"2024-04-26T02:21:18.617390+0000","modified":"2024-04-26T02:21:20.860445+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T02:21:20.860445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ea27580-6f9b-4e32-813f-fb8923bb6180","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8afe0d9-5e03-4ae1-b91d-54b1b7071d1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3a9234e-ef70-421d-b5ba-121f4f5cdcc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bfc148fb-9fd7-4a94-8287-52a4f38d7100","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3af86762-d77b-4c24-ac59-d0f20c4d423c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6a28bfec-7754-4a56-bbd1-0d1d2655fde7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"db087c2e-bd80-4863-a1db-5fbd8f9fe083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ccf8f7a9-c44f-44f4-91d1-9f197803f67d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:21:21.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.121+0000 7fbc3e02a640 1 -- 172.21.15.87:0/2821792017 >> v1:172.21.15.87:6790/0 conn(0x7fbc24004e70 legacy=0x7fbc381da340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:21.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.121+0000 7fbc3e02a640 1 -- 172.21.15.87:0/2821792017 shutdown_connections 2024-04-26T02:21:21.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.121+0000 7fbc3e02a640 1 -- 172.21.15.87:0/2821792017 >> 172.21.15.87:0/2821792017 conn(0x7fbc380b7a70 msgr2=0x7fbc380b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:21.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.121+0000 7fbc3e02a640 1 -- 172.21.15.87:0/2821792017 shutdown_connections 2024-04-26T02:21:21.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:21.121+0000 7fbc3e02a640 1 -- 172.21.15.87:0/2821792017 wait complete. 2024-04-26T02:21:21.143 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-26T02:21:21.144 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-26T02:21:21.389 INFO:tasks.ceph.mgr.y.smithi087.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-26T02:21:21.389 INFO:tasks.ceph.mgr.y.smithi087.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-26T02:21:21.389 INFO:tasks.ceph.mgr.y.smithi087.stderr: from numpy import show_config as show_numpy_config 2024-04-26T02:21:21.463 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:21.465+0000 7f024dae3180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T02:21:21.482 INFO:tasks.ceph.mgr.x.smithi122.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-26T02:21:21.482 INFO:tasks.ceph.mgr.x.smithi122.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-26T02:21:21.483 INFO:tasks.ceph.mgr.x.smithi122.stderr: from numpy import show_config as show_numpy_config 2024-04-26T02:21:21.487 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-04-26T02:21:21.489+0000 7f5a5a0a17c0 -1 Falling back to public interface 2024-04-26T02:21:21.491 INFO:tasks.ceph.osd.4.smithi122.stderr:2024-04-26T02:21:21.492+0000 7f59a12767c0 -1 Falling back to public interface 2024-04-26T02:21:21.502 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-04-26T02:21:21.505+0000 7f4db537b7c0 -1 Falling back to public interface 2024-04-26T02:21:21.514 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-04-26T02:21:21.517+0000 7fbee611d7c0 -1 Falling back to public interface 2024-04-26T02:21:21.519 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-04-26T02:21:21.521+0000 7f00f4a5d7c0 -1 Falling back to public interface 2024-04-26T02:21:21.519 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-04-26T02:21:21.520+0000 7f731ed0b7c0 -1 Falling back to public interface 2024-04-26T02:21:21.519 INFO:tasks.ceph.osd.7.smithi122.stderr:2024-04-26T02:21:21.520+0000 7fde67f497c0 -1 Falling back to public interface 2024-04-26T02:21:21.526 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:21.528+0000 7fc960988180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T02:21:21.543 INFO:tasks.ceph.osd.5.smithi122.stderr:2024-04-26T02:21:21.544+0000 7fdb379d57c0 -1 Falling back to public interface 2024-04-26T02:21:21.637 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:21.637+0000 7f024dae3180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T02:21:21.668 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:21.668+0000 7fc960988180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T02:21:21.791 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:21.793+0000 7f024dae3180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T02:21:21.834 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:21.836+0000 7fc960988180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T02:21:21.855 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:21.857+0000 7f024dae3180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T02:21:21.897 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:21.896+0000 7fc960988180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T02:21:21.949 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T02:21:21.988 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:21.989+0000 7f024dae3180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T02:21:22.030 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:22.028+0000 7fc960988180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T02:21:22.062 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73f6848640 1 Processor -- start 2024-04-26T02:21:22.062 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73f6848640 1 -- start start 2024-04-26T02:21:22.064 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73f6848640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f00a87c0 con 0x7f73f0140ab0 2024-04-26T02:21:22.064 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73f6848640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f00a8aa0 con 0x7f73f00b0b40 2024-04-26T02:21:22.065 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73f6848640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f00a8d80 con 0x7f73f013d900 2024-04-26T02:21:22.065 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73ef7fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f73f013d900 0x7f73f01403a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:55712/0 (socket says 172.21.15.87:55712) 2024-04-26T02:21:22.065 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73ef7fe640 1 -- 172.21.15.87:0/2674735072 learned_addr learned my addr 172.21.15.87:0/2674735072 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:22.065 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73eeffd640 1 -- 172.21.15.87:0/2674735072 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1936092784 0 0) 0x7f73f00a87c0 con 0x7f73f0140ab0 2024-04-26T02:21:22.065 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73eeffd640 1 -- 172.21.15.87:0/2674735072 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73d8003530 con 0x7f73f0140ab0 2024-04-26T02:21:22.065 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73eeffd640 1 -- 172.21.15.87:0/2674735072 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1152065584 0 0) 0x7f73f00a8d80 con 0x7f73f013d900 2024-04-26T02:21:22.065 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73eeffd640 1 -- 172.21.15.87:0/2674735072 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73f00a87c0 con 0x7f73f013d900 2024-04-26T02:21:22.065 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73eeffd640 1 -- 172.21.15.87:0/2674735072 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2107586125 0 0) 0x7f73d8003530 con 0x7f73f0140ab0 2024-04-26T02:21:22.065 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73eeffd640 1 -- 172.21.15.87:0/2674735072 >> v1:172.21.15.87:6790/0 conn(0x7f73f013d900 legacy=0x7f73f01403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:22.065 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73eeffd640 1 -- 172.21.15.87:0/2674735072 >> v1:172.21.15.122:6789/0 conn(0x7f73f00b0b40 legacy=0x7f73f013d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:22.065 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73eeffd640 1 -- 172.21.15.87:0/2674735072 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f00a9060 con 0x7f73f0140ab0 2024-04-26T02:21:22.065 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73eeffd640 1 -- 172.21.15.87:0/2674735072 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f73e80019a0 con 0x7f73f0140ab0 2024-04-26T02:21:22.065 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73eeffd640 1 -- 172.21.15.87:0/2674735072 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f73e8001c00 con 0x7f73f0140ab0 2024-04-26T02:21:22.065 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73eeffd640 1 -- 172.21.15.87:0/2674735072 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f73e8001f50 con 0x7f73f0140ab0 2024-04-26T02:21:22.066 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73f6848640 1 -- 172.21.15.87:0/2674735072 >> v1:172.21.15.87:6789/0 conn(0x7f73f0140ab0 legacy=0x7f73f014b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:22.066 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73f6848640 1 -- 172.21.15.87:0/2674735072 shutdown_connections 2024-04-26T02:21:22.066 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73f6848640 1 -- 172.21.15.87:0/2674735072 >> 172.21.15.87:0/2674735072 conn(0x7f73f00b5b10 msgr2=0x7f73f00b5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:22.066 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73f6848640 1 -- 172.21.15.87:0/2674735072 shutdown_connections 2024-04-26T02:21:22.066 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73f6848640 1 -- 172.21.15.87:0/2674735072 wait complete. 2024-04-26T02:21:22.068 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.065+0000 7f73f6848640 1 Processor -- start 2024-04-26T02:21:22.068 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73f6848640 1 -- start start 2024-04-26T02:21:22.069 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73f6848640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f01d77f0 con 0x7f73f00b0b40 2024-04-26T02:21:22.069 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73f6848640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f0139300 con 0x7f73f0140ab0 2024-04-26T02:21:22.069 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73f6848640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f01394d0 con 0x7f73f013d900 2024-04-26T02:21:22.069 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73effff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f73f00b0b40 0x7f73f01d8080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:35446/0 (socket says 172.21.15.87:35446) 2024-04-26T02:21:22.069 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73effff640 1 -- 172.21.15.87:0/2810760129 learned_addr learned my addr 172.21.15.87:0/2810760129 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:22.069 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1618719151 0 0) 0x7f73f01d77f0 con 0x7f73f00b0b40 2024-04-26T02:21:22.069 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73dc003530 con 0x7f73f00b0b40 2024-04-26T02:21:22.069 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 156303776 0 0) 0x7f73f01394d0 con 0x7f73f013d900 2024-04-26T02:21:22.069 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73f01d77f0 con 0x7f73f013d900 2024-04-26T02:21:22.069 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 603878641 0 0) 0x7f73f01d77f0 con 0x7f73f013d900 2024-04-26T02:21:22.069 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73f01394d0 con 0x7f73f013d900 2024-04-26T02:21:22.069 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3446761325 0 0) 0x7f73dc003530 con 0x7f73f00b0b40 2024-04-26T02:21:22.069 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73f01d77f0 con 0x7f73f00b0b40 2024-04-26T02:21:22.069 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f73e4001800 con 0x7f73f013d900 2024-04-26T02:21:22.070 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3860383882 0 0) 0x7f73f0139300 con 0x7f73f0140ab0 2024-04-26T02:21:22.070 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73dc003530 con 0x7f73f0140ab0 2024-04-26T02:21:22.070 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f73e0001900 con 0x7f73f00b0b40 2024-04-26T02:21:22.070 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4053292973 0 0) 0x7f73f01394d0 con 0x7f73f013d900 2024-04-26T02:21:22.070 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 >> v1:172.21.15.122:6789/0 conn(0x7f73f0140ab0 legacy=0x7f73f006d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:22.070 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 >> v1:172.21.15.87:6789/0 conn(0x7f73f00b0b40 legacy=0x7f73f01d8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:22.070 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f01396a0 con 0x7f73f013d900 2024-04-26T02:21:22.070 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73f6848640 1 -- 172.21.15.87:0/2810760129 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f73f013ac80 con 0x7f73f013d900 2024-04-26T02:21:22.070 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f73e40047f0 con 0x7f73f013d900 2024-04-26T02:21:22.070 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f73e4004bc0 con 0x7f73f013d900 2024-04-26T02:21:22.070 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73f6848640 1 -- 172.21.15.87:0/2810760129 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f73f006d780 con 0x7f73f013d900 2024-04-26T02:21:22.070 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f73e4005240 con 0x7f73f013d900 2024-04-26T02:21:22.070 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (unknown 16898136 0 0) 0x7f73e4006b80 con 0x7f73f013d900 2024-04-26T02:21:22.070 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.069+0000 7f73f6848640 1 -- 172.21.15.87:0/2810760129 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73f01d79c0 con 0x7f73f013d900 2024-04-26T02:21:22.071 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.073+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f73e40064b0 con 0x7f73f013d900 2024-04-26T02:21:22.129 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.129+0000 7f73f6848640 1 -- 172.21.15.87:0/2810760129 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73f0068390 con 0x7f73f013d900 2024-04-26T02:21:22.129 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.133+0000 7f73ecff9640 1 -- 172.21.15.87:0/2810760129 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (unknown 4040575075 0 37819316) 0x7f73e4006770 con 0x7f73f013d900 2024-04-26T02:21:22.130 INFO:teuthology.misc.health.smithi087.stdout: 2024-04-26T02:21:22.130 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"6232cbab-48ed-4660-9de8-286f4c0e9a32","created":"2024-04-26T02:21:18.617390+0000","modified":"2024-04-26T02:21:20.860445+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T02:21:20.860445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ea27580-6f9b-4e32-813f-fb8923bb6180","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b8afe0d9-5e03-4ae1-b91d-54b1b7071d1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3a9234e-ef70-421d-b5ba-121f4f5cdcc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bfc148fb-9fd7-4a94-8287-52a4f38d7100","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3af86762-d77b-4c24-ac59-d0f20c4d423c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6a28bfec-7754-4a56-bbd1-0d1d2655fde7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"db087c2e-bd80-4863-a1db-5fbd8f9fe083","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ccf8f7a9-c44f-44f4-91d1-9f197803f67d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:21:22.131 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.133+0000 7f73f6848640 1 -- 172.21.15.87:0/2810760129 >> v1:172.21.15.87:6790/0 conn(0x7f73f013d900 legacy=0x7f73f0138bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:22.131 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.133+0000 7f73f6848640 1 -- 172.21.15.87:0/2810760129 shutdown_connections 2024-04-26T02:21:22.131 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.133+0000 7f73f6848640 1 -- 172.21.15.87:0/2810760129 >> 172.21.15.87:0/2810760129 conn(0x7f73f00b5b10 msgr2=0x7f73f0074b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:22.131 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.133+0000 7f73f6848640 1 -- 172.21.15.87:0/2810760129 shutdown_connections 2024-04-26T02:21:22.131 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:22.133+0000 7f73f6848640 1 -- 172.21.15.87:0/2810760129 wait complete. 2024-04-26T02:21:22.144 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-26T02:21:22.629 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:22.629+0000 7f024dae3180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T02:21:22.630 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:22.632+0000 7fc960988180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T02:21:22.705 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:22.705+0000 7f024dae3180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T02:21:22.705 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:22.704+0000 7fc960988180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T02:21:22.886 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:22.885+0000 7f024dae3180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T02:21:22.886 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:22.884+0000 7fc960988180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T02:21:22.958 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:22.960+0000 7fc960988180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T02:21:22.959 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:22.961+0000 7f024dae3180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T02:21:23.243 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:23.245+0000 7f024dae3180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T02:21:23.244 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:23.244+0000 7fc960988180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T02:21:23.304 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:23.304+0000 7fc960988180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T02:21:23.305 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-04-26T02:21:23.305+0000 7f024dae3180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T02:21:24.344 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:24.344+0000 7fc907dd1640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-26T02:21:24.344 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-04-26T02:21:24.344+0000 7fc907dd1640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-26T02:21:24.590 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-04-26T02:21:24.588+0000 7f731ed0b7c0 -1 osd.6 0 log_to_monitors true 2024-04-26T02:21:25.286 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-04-26T02:21:25.289+0000 7fbee611d7c0 -1 osd.2 0 log_to_monitors true 2024-04-26T02:21:25.360 INFO:tasks.ceph.osd.7.smithi122.stderr:2024-04-26T02:21:25.360+0000 7fde67f497c0 -1 osd.7 0 log_to_monitors true 2024-04-26T02:21:25.406 INFO:tasks.ceph.osd.5.smithi122.stderr:2024-04-26T02:21:25.408+0000 7fdb379d57c0 -1 osd.5 0 log_to_monitors true 2024-04-26T02:21:25.414 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-04-26T02:21:25.417+0000 7f00f4a5d7c0 -1 osd.3 0 log_to_monitors true 2024-04-26T02:21:25.427 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-04-26T02:21:25.429+0000 7f5a5a0a17c0 -1 osd.1 0 log_to_monitors true 2024-04-26T02:21:25.440 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-04-26T02:21:25.441+0000 7f4db537b7c0 -1 osd.0 0 log_to_monitors true 2024-04-26T02:21:25.450 INFO:tasks.ceph.osd.4.smithi122.stderr:2024-04-26T02:21:25.452+0000 7f59a12767c0 -1 osd.4 0 log_to_monitors true 2024-04-26T02:21:26.363 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-04-26T02:21:26.365+0000 7fbee20c6640 -1 osd.2 0 waiting for initial osdmap 2024-04-26T02:21:26.372 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-04-26T02:21:26.373+0000 7fbeddf03640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T02:21:27.016 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-04-26T02:21:27.016+0000 7f731acb4640 -1 osd.6 0 waiting for initial osdmap 2024-04-26T02:21:27.019 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-04-26T02:21:27.020+0000 7f7312ae9640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T02:21:27.378 INFO:tasks.ceph.osd.5.smithi122.stderr:2024-04-26T02:21:27.380+0000 7fdb3397e640 -1 osd.5 0 waiting for initial osdmap 2024-04-26T02:21:27.378 INFO:tasks.ceph.osd.7.smithi122.stderr:2024-04-26T02:21:27.380+0000 7fde63ef2640 -1 osd.7 0 waiting for initial osdmap 2024-04-26T02:21:27.382 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-04-26T02:21:27.385+0000 7f4db1324640 -1 osd.0 0 waiting for initial osdmap 2024-04-26T02:21:27.386 INFO:tasks.ceph.osd.7.smithi122.stderr:2024-04-26T02:21:27.388+0000 7fde5fd2f640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T02:21:27.387 INFO:tasks.ceph.osd.5.smithi122.stderr:2024-04-26T02:21:27.388+0000 7fdb2f7bb640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T02:21:27.393 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-04-26T02:21:27.397+0000 7f4dad161640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T02:21:28.039 INFO:tasks.ceph.osd.4.smithi122.stderr:2024-04-26T02:21:28.040+0000 7f599da31640 -1 osd.4 0 waiting for initial osdmap 2024-04-26T02:21:28.045 INFO:tasks.ceph.osd.4.smithi122.stderr:2024-04-26T02:21:28.044+0000 7f5995054640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T02:21:28.224 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-04-26T02:21:28.225+0000 7f00f0a06640 -1 osd.3 0 waiting for initial osdmap 2024-04-26T02:21:28.230 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-04-26T02:21:28.233+0000 7f00e883b640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T02:21:28.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-04-26T02:21:28.253+0000 7f5a5685c640 -1 osd.1 0 waiting for initial osdmap 2024-04-26T02:21:28.254 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-04-26T02:21:28.257+0000 7f5a4de7f640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T02:21:28.951 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T02:21:29.073 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.073+0000 7f9052cf9640 1 Processor -- start 2024-04-26T02:21:29.073 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.073+0000 7f9052cf9640 1 -- start start 2024-04-26T02:21:29.075 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f9052cf9640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904c0a8c80 con 0x7f904c0ae9d0 2024-04-26T02:21:29.075 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f9052cf9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904c0a8f60 con 0x7f904c149810 2024-04-26T02:21:29.075 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f9052cf9640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904c0af810 con 0x7f904c0aedb0 2024-04-26T02:21:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f904bfff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f904c0aedb0 0x7f904c13d2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40120/0 (socket says 172.21.15.87:40120) 2024-04-26T02:21:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f904bfff640 1 -- 172.21.15.87:0/4141942257 learned_addr learned my addr 172.21.15.87:0/4141942257 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f904b7fe640 1 -- 172.21.15.87:0/4141942257 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4157878494 0 0) 0x7f904c0af810 con 0x7f904c0aedb0 2024-04-26T02:21:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f904b7fe640 1 -- 172.21.15.87:0/4141942257 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9038003530 con 0x7f904c0aedb0 2024-04-26T02:21:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f904b7fe640 1 -- 172.21.15.87:0/4141942257 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 362013037 0 0) 0x7f904c0a8c80 con 0x7f904c0ae9d0 2024-04-26T02:21:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f904b7fe640 1 -- 172.21.15.87:0/4141942257 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f904c0af810 con 0x7f904c0ae9d0 2024-04-26T02:21:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f904b7fe640 1 -- 172.21.15.87:0/4141942257 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3131135705 0 0) 0x7f9038003530 con 0x7f904c0aedb0 2024-04-26T02:21:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f904b7fe640 1 -- 172.21.15.87:0/4141942257 >> v1:172.21.15.122:6789/0 conn(0x7f904c149810 legacy=0x7f904c14bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f904b7fe640 1 -- 172.21.15.87:0/4141942257 >> v1:172.21.15.87:6789/0 conn(0x7f904c0ae9d0 legacy=0x7f904c143ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f904b7fe640 1 -- 172.21.15.87:0/4141942257 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904c1435b0 con 0x7f904c0aedb0 2024-04-26T02:21:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f904b7fe640 1 -- 172.21.15.87:0/4141942257 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f90440019e0 con 0x7f904c0aedb0 2024-04-26T02:21:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f904b7fe640 1 -- 172.21.15.87:0/4141942257 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9044001c40 con 0x7f904c0aedb0 2024-04-26T02:21:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f904b7fe640 1 -- 172.21.15.87:0/4141942257 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9044002010 con 0x7f904c0aedb0 2024-04-26T02:21:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f9052cf9640 1 -- 172.21.15.87:0/4141942257 >> v1:172.21.15.87:6790/0 conn(0x7f904c0aedb0 legacy=0x7f904c13d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f9052cf9640 1 -- 172.21.15.87:0/4141942257 shutdown_connections 2024-04-26T02:21:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f9052cf9640 1 -- 172.21.15.87:0/4141942257 >> 172.21.15.87:0/4141942257 conn(0x7f904c06b9d0 msgr2=0x7f904c0b3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f9052cf9640 1 -- 172.21.15.87:0/4141942257 shutdown_connections 2024-04-26T02:21:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f9052cf9640 1 -- 172.21.15.87:0/4141942257 wait complete. 2024-04-26T02:21:29.079 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f9052cf9640 1 Processor -- start 2024-04-26T02:21:29.079 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.077+0000 7f9052cf9640 1 -- start start 2024-04-26T02:21:29.080 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f9052cf9640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904c140af0 con 0x7f904c0aedb0 2024-04-26T02:21:29.080 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f9052cf9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904c1d5f30 con 0x7f904c149810 2024-04-26T02:21:29.080 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f9052cf9640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904c1d5300 con 0x7f904c0ae9d0 2024-04-26T02:21:29.080 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f9050a95640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f904c0ae9d0 0x7f904c13f2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40122/0 (socket says 172.21.15.87:40122) 2024-04-26T02:21:29.080 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f9050a95640 1 -- 172.21.15.87:0/3369113105 learned_addr learned my addr 172.21.15.87:0/3369113105 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:29.080 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3307921946 0 0) 0x7f904c140af0 con 0x7f904c0aedb0 2024-04-26T02:21:29.080 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f902c003530 con 0x7f904c0aedb0 2024-04-26T02:21:29.080 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2724236076 0 0) 0x7f904c1d5300 con 0x7f904c0ae9d0 2024-04-26T02:21:29.080 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f904c140af0 con 0x7f904c0ae9d0 2024-04-26T02:21:29.080 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3673298660 0 0) 0x7f904c140af0 con 0x7f904c0ae9d0 2024-04-26T02:21:29.081 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f904c1d5300 con 0x7f904c0ae9d0 2024-04-26T02:21:29.081 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f903c001980 con 0x7f904c0ae9d0 2024-04-26T02:21:29.081 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3976714101 0 0) 0x7f902c003530 con 0x7f904c0aedb0 2024-04-26T02:21:29.081 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f904c140af0 con 0x7f904c0aedb0 2024-04-26T02:21:29.081 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9044001f90 con 0x7f904c0aedb0 2024-04-26T02:21:29.081 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2643024788 0 0) 0x7f904c1d5f30 con 0x7f904c149810 2024-04-26T02:21:29.081 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f902c003530 con 0x7f904c149810 2024-04-26T02:21:29.081 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1255969830 0 0) 0x7f904c1d5300 con 0x7f904c0ae9d0 2024-04-26T02:21:29.081 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 >> v1:172.21.15.122:6789/0 conn(0x7f904c149810 legacy=0x7f904c133ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:29.081 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 >> v1:172.21.15.87:6789/0 conn(0x7f904c0aedb0 legacy=0x7f904c1d5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:29.081 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904c13e680 con 0x7f904c0ae9d0 2024-04-26T02:21:29.081 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f9052cf9640 1 -- 172.21.15.87:0/3369113105 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f904c13e910 con 0x7f904c0ae9d0 2024-04-26T02:21:29.081 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f9052cf9640 1 -- 172.21.15.87:0/3369113105 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f904c138790 con 0x7f904c0ae9d0 2024-04-26T02:21:29.081 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f903c0036a0 con 0x7f904c0ae9d0 2024-04-26T02:21:29.082 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f903c003a70 con 0x7f904c0ae9d0 2024-04-26T02:21:29.082 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f9052cf9640 1 -- 172.21.15.87:0/3369113105 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9030005090 con 0x7f904c0ae9d0 2024-04-26T02:21:29.082 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) ==== 87354+0+0 (unknown 4190153105 0 0) 0x7f903c003c90 con 0x7f904c0ae9d0 2024-04-26T02:21:29.082 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.081+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(14..14 src has 1..14) ==== 3726+0+0 (unknown 39486977 0 0) 0x7f903c080e50 con 0x7f904c0ae9d0 2024-04-26T02:21:29.085 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.085+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f903c052200 con 0x7f904c0ae9d0 2024-04-26T02:21:29.246 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.249+0000 7f9052cf9640 1 -- 172.21.15.87:0/3369113105 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9030005380 con 0x7f904c0ae9d0 2024-04-26T02:21:29.246 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.249+0000 7f90497fa640 1 -- 172.21.15.87:0/3369113105 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+10762 (unknown 92231148 0 723060093) 0x7f903c051b00 con 0x7f904c0ae9d0 2024-04-26T02:21:29.247 INFO:teuthology.misc.health.smithi087.stdout: 2024-04-26T02:21:29.247 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":14,"fsid":"6232cbab-48ed-4660-9de8-286f4c0e9a32","created":"2024-04-26T02:21:18.617390+0000","modified":"2024-04-26T02:21:28.375870+0000","last_up_change":"2024-04-26T02:21:28.375870+0000","last_in_change":"2024-04-26T02:21:20.860445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T02:21:27.425849+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6ea27580-6f9b-4e32-813f-fb8923bb6180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6804","nonce":815800445}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6805","nonce":815800445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6807","nonce":815800445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6806","nonce":815800445}]},"public_addr":"172.21.15.87:6804/815800445","cluster_addr":"172.21.15.87:6805/815800445","heartbeat_back_addr":"172.21.15.87:6807/815800445","heartbeat_front_addr":"172.21.15.87:6806/815800445","state":["exists","up"]},{"osd":1,"uuid":"b8afe0d9-5e03-4ae1-b91d-54b1b7071d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6800","nonce":174846219}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6801","nonce":174846219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6803","nonce":174846219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6802","nonce":174846219}]},"public_addr":"172.21.15.87:6800/174846219","cluster_addr":"172.21.15.87:6801/174846219","heartbeat_back_addr":"172.21.15.87:6803/174846219","heartbeat_front_addr":"172.21.15.87:6802/174846219","state":["exists","up"]},{"osd":2,"uuid":"d3a9234e-ef70-421d-b5ba-121f4f5cdcc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6808","nonce":2992851618}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6809","nonce":2992851618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6811","nonce":2992851618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6810","nonce":2992851618}]},"public_addr":"172.21.15.87:6808/2992851618","cluster_addr":"172.21.15.87:6809/2992851618","heartbeat_back_addr":"172.21.15.87:6811/2992851618","heartbeat_front_addr":"172.21.15.87:6810/2992851618","state":["exists","up"]},{"osd":3,"uuid":"bfc148fb-9fd7-4a94-8287-52a4f38d7100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6812","nonce":3926229128}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6813","nonce":3926229128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6815","nonce":3926229128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6814","nonce":3926229128}]},"public_addr":"172.21.15.87:6812/3926229128","cluster_addr":"172.21.15.87:6813/3926229128","heartbeat_back_addr":"172.21.15.87:6815/3926229128","heartbeat_front_addr":"172.21.15.87:6814/3926229128","state":["exists","up"]},{"osd":4,"uuid":"3af86762-d77b-4c24-ac59-d0f20c4d423c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6800","nonce":850918874}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6801","nonce":850918874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6803","nonce":850918874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6802","nonce":850918874}]},"public_addr":"172.21.15.122:6800/850918874","cluster_addr":"172.21.15.122:6801/850918874","heartbeat_back_addr":"172.21.15.122:6803/850918874","heartbeat_front_addr":"172.21.15.122:6802/850918874","state":["exists","up"]},{"osd":5,"uuid":"6a28bfec-7754-4a56-bbd1-0d1d2655fde7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6812","nonce":498384109}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6813","nonce":498384109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6815","nonce":498384109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6814","nonce":498384109}]},"public_addr":"172.21.15.122:6812/498384109","cluster_addr":"172.21.15.122:6813/498384109","heartbeat_back_addr":"172.21.15.122:6815/498384109","heartbeat_front_addr":"172.21.15.122:6814/498384109","state":["exists","up"]},{"osd":6,"uuid":"db087c2e-bd80-4863-a1db-5fbd8f9fe083","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6805","nonce":662791991}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6807","nonce":662791991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6811","nonce":662791991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6810","nonce":662791991}]},"public_addr":"172.21.15.122:6805/662791991","cluster_addr":"172.21.15.122:6807/662791991","heartbeat_back_addr":"172.21.15.122:6811/662791991","heartbeat_front_addr":"172.21.15.122:6810/662791991","state":["exists","up"]},{"osd":7,"uuid":"ccf8f7a9-c44f-44f4-91d1-9f197803f67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6804","nonce":7687745}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6806","nonce":7687745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6809","nonce":7687745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6808","nonce":7687745}]},"public_addr":"172.21.15.122:6804/7687745","cluster_addr":"172.21.15.122:6806/7687745","heartbeat_back_addr":"172.21.15.122:6809/7687745","heartbeat_front_addr":"172.21.15.122:6808/7687745","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.335478+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:25.564818+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:21:29.249 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.249+0000 7f9052cf9640 1 -- 172.21.15.87:0/3369113105 >> v1:172.21.15.122:6816/416193625 conn(0x7f902c068380 legacy=0x7f902c06a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:29.249 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.249+0000 7f9052cf9640 1 -- 172.21.15.87:0/3369113105 >> v1:172.21.15.87:6790/0 conn(0x7f904c0ae9d0 legacy=0x7f904c13f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:29.249 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.249+0000 7f9052cf9640 1 -- 172.21.15.87:0/3369113105 shutdown_connections 2024-04-26T02:21:29.249 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.249+0000 7f9052cf9640 1 -- 172.21.15.87:0/3369113105 >> 172.21.15.87:0/3369113105 conn(0x7f904c06b9d0 msgr2=0x7f904c13d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:29.249 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.249+0000 7f9052cf9640 1 -- 172.21.15.87:0/3369113105 shutdown_connections 2024-04-26T02:21:29.249 INFO:teuthology.misc.health.smithi087.stderr:2024-04-26T02:21:29.249+0000 7f9052cf9640 1 -- 172.21.15.87:0/3369113105 wait complete. 2024-04-26T02:21:29.264 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-26T02:21:29.264 INFO:tasks.ceph:Creating RBD pool 2024-04-26T02:21:29.264 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-26T02:21:29.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.389+0000 7fd9cf190640 1 Processor -- start 2024-04-26T02:21:29.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.389+0000 7fd9cf190640 1 -- start start 2024-04-26T02:21:29.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.389+0000 7fd9cf190640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c80b11c0 con 0x7fd9c8143d70 2024-04-26T02:21:29.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.389+0000 7fd9cf190640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c80b06a0 con 0x7fd9c80b1920 2024-04-26T02:21:29.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.389+0000 7fd9cf190640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c80b0870 con 0x7fd9c80afd20 2024-04-26T02:21:29.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.389+0000 7fd9cd72d640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fd9c80afd20 0x7fd9c8151d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40132/0 (socket says 172.21.15.87:40132) 2024-04-26T02:21:29.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.389+0000 7fd9cd72d640 1 -- 172.21.15.87:0/3982746933 learned_addr learned my addr 172.21.15.87:0/3982746933 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:29.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.393+0000 7fd9c77fe640 1 -- 172.21.15.87:0/3982746933 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 701469427 0 0) 0x7fd9c80b11c0 con 0x7fd9c8143d70 2024-04-26T02:21:29.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.393+0000 7fd9c77fe640 1 -- 172.21.15.87:0/3982746933 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9b0003530 con 0x7fd9c8143d70 2024-04-26T02:21:29.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.393+0000 7fd9c77fe640 1 -- 172.21.15.87:0/3982746933 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1964991679 0 0) 0x7fd9b0003530 con 0x7fd9c8143d70 2024-04-26T02:21:29.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.393+0000 7fd9c77fe640 1 -- 172.21.15.87:0/3982746933 >> v1:172.21.15.87:6790/0 conn(0x7fd9c80afd20 legacy=0x7fd9c8151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:29.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.393+0000 7fd9c77fe640 1 -- 172.21.15.87:0/3982746933 >> v1:172.21.15.122:6789/0 conn(0x7fd9c80b1920 legacy=0x7fd9c8143660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:29.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.393+0000 7fd9c77fe640 1 -- 172.21.15.87:0/3982746933 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c806d7f0 con 0x7fd9c8143d70 2024-04-26T02:21:29.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.393+0000 7fd9c77fe640 1 -- 172.21.15.87:0/3982746933 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fd9b8001550 con 0x7fd9c8143d70 2024-04-26T02:21:29.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.393+0000 7fd9c77fe640 1 -- 172.21.15.87:0/3982746933 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9b80018b0 con 0x7fd9c8143d70 2024-04-26T02:21:29.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.393+0000 7fd9c77fe640 1 -- 172.21.15.87:0/3982746933 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fd9b8001c80 con 0x7fd9c8143d70 2024-04-26T02:21:29.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.393+0000 7fd9cf190640 1 -- 172.21.15.87:0/3982746933 >> v1:172.21.15.87:6789/0 conn(0x7fd9c8143d70 legacy=0x7fd9c806d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:29.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.393+0000 7fd9cf190640 1 -- 172.21.15.87:0/3982746933 shutdown_connections 2024-04-26T02:21:29.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.393+0000 7fd9cf190640 1 -- 172.21.15.87:0/3982746933 >> 172.21.15.87:0/3982746933 conn(0x7fd9c8067940 msgr2=0x7fd9c80b59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:29.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.393+0000 7fd9cf190640 1 -- 172.21.15.87:0/3982746933 shutdown_connections 2024-04-26T02:21:29.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.393+0000 7fd9cf190640 1 -- 172.21.15.87:0/3982746933 wait complete. 2024-04-26T02:21:29.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9cf190640 1 Processor -- start 2024-04-26T02:21:29.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9cf190640 1 -- start start 2024-04-26T02:21:29.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9cf190640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c813e510 con 0x7fd9c8143d70 2024-04-26T02:21:29.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9cf190640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c813e7f0 con 0x7fd9c80afd20 2024-04-26T02:21:29.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9cf190640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c813ead0 con 0x7fd9c80b1920 2024-04-26T02:21:29.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c7fff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fd9c80b1920 0x7fd9c813d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40146/0 (socket says 172.21.15.87:40146) 2024-04-26T02:21:29.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9cd72d640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fd9c8143d70 0x7fd9c81e6610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53538/0 (socket says 172.21.15.87:53538) 2024-04-26T02:21:29.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c7fff640 1 -- 172.21.15.87:0/2569819780 learned_addr learned my addr 172.21.15.87:0/2569819780 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:29.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4093907968 0 0) 0x7fd9c813e510 con 0x7fd9c8143d70 2024-04-26T02:21:29.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9b4003530 con 0x7fd9c8143d70 2024-04-26T02:21:29.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2149764708 0 0) 0x7fd9c813ead0 con 0x7fd9c80b1920 2024-04-26T02:21:29.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9c813e510 con 0x7fd9c80b1920 2024-04-26T02:21:29.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3314092072 0 0) 0x7fd9b4003530 con 0x7fd9c8143d70 2024-04-26T02:21:29.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9c813ead0 con 0x7fd9c8143d70 2024-04-26T02:21:29.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3235827129 0 0) 0x7fd9c813e510 con 0x7fd9c80b1920 2024-04-26T02:21:29.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9b4003530 con 0x7fd9c80b1920 2024-04-26T02:21:29.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fd9c0001b30 con 0x7fd9c8143d70 2024-04-26T02:21:29.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 57432035 0 0) 0x7fd9c813e7f0 con 0x7fd9c80afd20 2024-04-26T02:21:29.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9c813e510 con 0x7fd9c80afd20 2024-04-26T02:21:29.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fd9b8001610 con 0x7fd9c80b1920 2024-04-26T02:21:29.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1106744286 0 0) 0x7fd9c813ead0 con 0x7fd9c8143d70 2024-04-26T02:21:29.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 >> v1:172.21.15.87:6790/0 conn(0x7fd9c80b1920 legacy=0x7fd9c813d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:29.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 >> v1:172.21.15.122:6789/0 conn(0x7fd9c80afd20 legacy=0x7fd9c813eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:29.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c813ce80 con 0x7fd9c8143d70 2024-04-26T02:21:29.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9cf190640 1 -- 172.21.15.87:0/2569819780 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c813c470 con 0x7fd9c8143d70 2024-04-26T02:21:29.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9cf190640 1 -- 172.21.15.87:0/2569819780 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd9c8149700 con 0x7fd9c8143d70 2024-04-26T02:21:29.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9c0004b70 con 0x7fd9c8143d70 2024-04-26T02:21:29.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fd9c0004ec0 con 0x7fd9c8143d70 2024-04-26T02:21:29.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.397+0000 7fd9cf190640 1 -- 172.21.15.87:0/2569819780 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9a8005090 con 0x7fd9c8143d70 2024-04-26T02:21:29.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.401+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) ==== 87354+0+0 (unknown 4190153105 0 0) 0x7fd9c001a5b0 con 0x7fd9c8143d70 2024-04-26T02:21:29.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.401+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(15..15 src has 1..15) ==== 3737+0+0 (unknown 3468363777 0 0) 0x7fd9c0081030 con 0x7fd9c8143d70 2024-04-26T02:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.405+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fd9c00522f0 con 0x7fd9c8143d70 2024-04-26T02:21:29.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:29.577+0000 7fd9cf190640 1 -- 172.21.15.87:0/2569819780 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fd9a8005b90 con 0x7fd9c8143d70 2024-04-26T02:21:30.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.401+0000 7fd9c57fa640 1 -- 172.21.15.87:0/2569819780 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) ==== 109+0+0 (unknown 231814880 0 0) 0x7fd9c0051bf0 con 0x7fd9c8143d70 2024-04-26T02:21:30.398 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'rbd' created 2024-04-26T02:21:30.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.405+0000 7fd9cf190640 1 -- 172.21.15.87:0/2569819780 >> v1:172.21.15.122:6816/416193625 conn(0x7fd9b40683b0 legacy=0x7fd9b406a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:30.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.405+0000 7fd9cf190640 1 -- 172.21.15.87:0/2569819780 >> v1:172.21.15.87:6789/0 conn(0x7fd9c8143d70 legacy=0x7fd9c81e6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:30.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.405+0000 7fd9cf190640 1 -- 172.21.15.87:0/2569819780 shutdown_connections 2024-04-26T02:21:30.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.405+0000 7fd9cf190640 1 -- 172.21.15.87:0/2569819780 >> 172.21.15.87:0/2569819780 conn(0x7fd9c8067940 msgr2=0x7fd9c8143430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:30.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.405+0000 7fd9cf190640 1 -- 172.21.15.87:0/2569819780 shutdown_connections 2024-04-26T02:21:30.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.405+0000 7fd9cf190640 1 -- 172.21.15.87:0/2569819780 wait complete. 2024-04-26T02:21:30.423 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-26T02:21:30.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.549+0000 7f4d37bb7640 1 Processor -- start 2024-04-26T02:21:30.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.549+0000 7f4d37bb7640 1 -- start start 2024-04-26T02:21:30.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.549+0000 7f4d37bb7640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d300a8720 con 0x7f4d3013d270 2024-04-26T02:21:30.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.549+0000 7f4d37bb7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d300a8a00 con 0x7f4d30143c60 2024-04-26T02:21:30.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.549+0000 7f4d37bb7640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d300a8ce0 con 0x7f4d300b4200 2024-04-26T02:21:30.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d35953640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f4d300b4200 0x7f4d30143550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40154/0 (socket says 172.21.15.87:40154) 2024-04-26T02:21:30.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d35953640 1 -- 172.21.15.87:0/3187226174 learned_addr learned my addr 172.21.15.87:0/3187226174 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:30.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d34951640 1 -- 172.21.15.87:0/3187226174 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2691065692 0 0) 0x7f4d300a8ce0 con 0x7f4d300b4200 2024-04-26T02:21:30.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d34951640 1 -- 172.21.15.87:0/3187226174 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d1c003530 con 0x7f4d300b4200 2024-04-26T02:21:30.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d34951640 1 -- 172.21.15.87:0/3187226174 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3715120571 0 0) 0x7f4d300a8720 con 0x7f4d3013d270 2024-04-26T02:21:30.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d34951640 1 -- 172.21.15.87:0/3187226174 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d300a8ce0 con 0x7f4d3013d270 2024-04-26T02:21:30.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d34951640 1 -- 172.21.15.87:0/3187226174 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2717652133 0 0) 0x7f4d1c003530 con 0x7f4d300b4200 2024-04-26T02:21:30.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d34951640 1 -- 172.21.15.87:0/3187226174 >> v1:172.21.15.122:6789/0 conn(0x7f4d30143c60 legacy=0x7f4d3013cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:30.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d34951640 1 -- 172.21.15.87:0/3187226174 >> v1:172.21.15.87:6789/0 conn(0x7f4d3013d270 legacy=0x7f4d3014bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:30.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d34951640 1 -- 172.21.15.87:0/3187226174 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d300a8fc0 con 0x7f4d300b4200 2024-04-26T02:21:30.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d34951640 1 -- 172.21.15.87:0/3187226174 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f4d2c001620 con 0x7f4d300b4200 2024-04-26T02:21:30.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d34951640 1 -- 172.21.15.87:0/3187226174 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4d2c001850 con 0x7f4d300b4200 2024-04-26T02:21:30.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d34951640 1 -- 172.21.15.87:0/3187226174 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f4d2c001c60 con 0x7f4d300b4200 2024-04-26T02:21:30.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d37bb7640 1 -- 172.21.15.87:0/3187226174 >> v1:172.21.15.87:6790/0 conn(0x7f4d300b4200 legacy=0x7f4d30143550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:30.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d37bb7640 1 -- 172.21.15.87:0/3187226174 shutdown_connections 2024-04-26T02:21:30.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d37bb7640 1 -- 172.21.15.87:0/3187226174 >> 172.21.15.87:0/3187226174 conn(0x7f4d3006b920 msgr2=0x7f4d300b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:30.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d37bb7640 1 -- 172.21.15.87:0/3187226174 shutdown_connections 2024-04-26T02:21:30.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d37bb7640 1 -- 172.21.15.87:0/3187226174 wait complete. 2024-04-26T02:21:30.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d37bb7640 1 Processor -- start 2024-04-26T02:21:30.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d37bb7640 1 -- start start 2024-04-26T02:21:30.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d37bb7640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d30140ad0 con 0x7f4d3013d270 2024-04-26T02:21:30.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d37bb7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d30136530 con 0x7f4d30143c60 2024-04-26T02:21:30.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d37bb7640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d30135a90 con 0x7f4d300b4200 2024-04-26T02:21:30.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d35953640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f4d300b4200 0x7f4d301d75c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40168/0 (socket says 172.21.15.87:40168) 2024-04-26T02:21:30.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d35953640 1 -- 172.21.15.87:0/99748832 learned_addr learned my addr 172.21.15.87:0/99748832 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:30.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d35152640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f4d3013d270 0x7f4d3013e6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53568/0 (socket says 172.21.15.87:53568) 2024-04-26T02:21:30.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1475041547 0 0) 0x7f4d30135a90 con 0x7f4d300b4200 2024-04-26T02:21:30.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d14003530 con 0x7f4d300b4200 2024-04-26T02:21:30.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 207135036 0 0) 0x7f4d30140ad0 con 0x7f4d3013d270 2024-04-26T02:21:30.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d30135a90 con 0x7f4d3013d270 2024-04-26T02:21:30.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3647052175 0 0) 0x7f4d14003530 con 0x7f4d300b4200 2024-04-26T02:21:30.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4d30140ad0 con 0x7f4d300b4200 2024-04-26T02:21:30.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3523928550 0 0) 0x7f4d30136530 con 0x7f4d30143c60 2024-04-26T02:21:30.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d14003530 con 0x7f4d30143c60 2024-04-26T02:21:30.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3495585013 0 0) 0x7f4d30135a90 con 0x7f4d3013d270 2024-04-26T02:21:30.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4d30136530 con 0x7f4d3013d270 2024-04-26T02:21:30.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f4d2c001c60 con 0x7f4d300b4200 2024-04-26T02:21:30.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.553+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f4d240017f0 con 0x7f4d3013d270 2024-04-26T02:21:30.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.557+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1867486789 0 0) 0x7f4d30140ad0 con 0x7f4d300b4200 2024-04-26T02:21:30.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.557+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 >> v1:172.21.15.122:6789/0 conn(0x7f4d30143c60 legacy=0x7f4d3013f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:30.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.557+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 >> v1:172.21.15.87:6789/0 conn(0x7f4d3013d270 legacy=0x7f4d3013e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:30.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.557+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d30135c60 con 0x7f4d300b4200 2024-04-26T02:21:30.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.557+0000 7f4d37bb7640 1 -- 172.21.15.87:0/99748832 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4d301e3010 con 0x7f4d300b4200 2024-04-26T02:21:30.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.557+0000 7f4d37bb7640 1 -- 172.21.15.87:0/99748832 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4d301e33b0 con 0x7f4d300b4200 2024-04-26T02:21:30.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.557+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4d2c004c00 con 0x7f4d300b4200 2024-04-26T02:21:30.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.557+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f4d2c004fd0 con 0x7f4d300b4200 2024-04-26T02:21:30.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.557+0000 7f4d37bb7640 1 -- 172.21.15.87:0/99748832 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d18005090 con 0x7f4d300b4200 2024-04-26T02:21:30.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.557+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) ==== 87354+0+0 (unknown 4190153105 0 0) 0x7f4d2c005170 con 0x7f4d300b4200 2024-04-26T02:21:30.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.557+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(16..16 src has 1..16) ==== 4079+0+0 (unknown 2271699357 0 0) 0x7f4d2c0810d0 con 0x7f4d300b4200 2024-04-26T02:21:30.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.561+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f4d2c0521e0 con 0x7f4d300b4200 2024-04-26T02:21:30.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:30.725+0000 7f4d37bb7640 1 -- 172.21.15.87:0/99748832 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) -- 0x7f4d18005380 con 0x7f4d300b4200 2024-04-26T02:21:31.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.421+0000 7f4d227fc640 1 -- 172.21.15.87:0/99748832 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v17) ==== 173+0+0 (unknown 2255673937 0 0) 0x7f4d2c051ae0 con 0x7f4d300b4200 2024-04-26T02:21:31.421 INFO:teuthology.orchestra.run.smithi087.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-26T02:21:31.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.425+0000 7f4d37bb7640 1 -- 172.21.15.87:0/99748832 >> v1:172.21.15.122:6816/416193625 conn(0x7f4d140682e0 legacy=0x7f4d1406a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:31.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.425+0000 7f4d37bb7640 1 -- 172.21.15.87:0/99748832 >> v1:172.21.15.87:6790/0 conn(0x7f4d300b4200 legacy=0x7f4d301d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:31.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.425+0000 7f4d37bb7640 1 -- 172.21.15.87:0/99748832 shutdown_connections 2024-04-26T02:21:31.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.425+0000 7f4d37bb7640 1 -- 172.21.15.87:0/99748832 >> 172.21.15.87:0/99748832 conn(0x7f4d3006b920 msgr2=0x7f4d300b23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:31.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.425+0000 7f4d37bb7640 1 -- 172.21.15.87:0/99748832 shutdown_connections 2024-04-26T02:21:31.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.425+0000 7f4d37bb7640 1 -- 172.21.15.87:0/99748832 wait complete. 2024-04-26T02:21:31.446 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-26T02:21:31.447 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T02:21:31.447 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-26T02:21:31.447 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-26T02:21:31.448 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-26T02:21:31.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.565+0000 7f8abec19640 1 Processor -- start 2024-04-26T02:21:31.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.565+0000 7f8abec19640 1 -- start start 2024-04-26T02:21:31.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.565+0000 7f8abec19640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ab80ab1a0 con 0x7f8ab80a9e60 2024-04-26T02:21:31.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.565+0000 7f8abec19640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ab80a9a10 con 0x7f8ab80a8bf0 2024-04-26T02:21:31.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.565+0000 7f8abec19640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ab80a8fd0 con 0x7f8ab813f9e0 2024-04-26T02:21:31.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.565+0000 7f8ab7fff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f8ab80a9e60 0x7f8ab8145980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53586/0 (socket says 172.21.15.87:53586) 2024-04-26T02:21:31.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.565+0000 7f8ab7fff640 1 -- 172.21.15.87:0/638982820 learned_addr learned my addr 172.21.15.87:0/638982820 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:31.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab77fe640 1 -- 172.21.15.87:0/638982820 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1799358141 0 0) 0x7f8ab80ab1a0 con 0x7f8ab80a9e60 2024-04-26T02:21:31.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab77fe640 1 -- 172.21.15.87:0/638982820 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8aa0003530 con 0x7f8ab80a9e60 2024-04-26T02:21:31.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab77fe640 1 -- 172.21.15.87:0/638982820 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3756457908 0 0) 0x7f8ab80a8fd0 con 0x7f8ab813f9e0 2024-04-26T02:21:31.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab77fe640 1 -- 172.21.15.87:0/638982820 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ab80ab1a0 con 0x7f8ab813f9e0 2024-04-26T02:21:31.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab77fe640 1 -- 172.21.15.87:0/638982820 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4020710071 0 0) 0x7f8aa0003530 con 0x7f8ab80a9e60 2024-04-26T02:21:31.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab77fe640 1 -- 172.21.15.87:0/638982820 >> v1:172.21.15.87:6790/0 conn(0x7f8ab813f9e0 legacy=0x7f8ab814d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:31.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab77fe640 1 -- 172.21.15.87:0/638982820 >> v1:172.21.15.122:6789/0 conn(0x7f8ab80a8bf0 legacy=0x7f8ab813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:31.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab77fe640 1 -- 172.21.15.87:0/638982820 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ab80a91a0 con 0x7f8ab80a9e60 2024-04-26T02:21:31.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab77fe640 1 -- 172.21.15.87:0/638982820 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8aac001620 con 0x7f8ab80a9e60 2024-04-26T02:21:31.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab77fe640 1 -- 172.21.15.87:0/638982820 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8aac001890 con 0x7f8ab80a9e60 2024-04-26T02:21:31.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab77fe640 1 -- 172.21.15.87:0/638982820 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8aac001c60 con 0x7f8ab80a9e60 2024-04-26T02:21:31.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8abec19640 1 -- 172.21.15.87:0/638982820 >> v1:172.21.15.87:6789/0 conn(0x7f8ab80a9e60 legacy=0x7f8ab8145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:31.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8abec19640 1 -- 172.21.15.87:0/638982820 shutdown_connections 2024-04-26T02:21:31.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8abec19640 1 -- 172.21.15.87:0/638982820 >> 172.21.15.87:0/638982820 conn(0x7f8ab80b7af0 msgr2=0x7f8ab80b7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:31.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8abec19640 1 -- 172.21.15.87:0/638982820 shutdown_connections 2024-04-26T02:21:31.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8abec19640 1 -- 172.21.15.87:0/638982820 wait complete. 2024-04-26T02:21:31.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8abec19640 1 Processor -- start 2024-04-26T02:21:31.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8abec19640 1 -- start start 2024-04-26T02:21:31.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8abec19640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ab8144ce0 con 0x7f8ab813f9e0 2024-04-26T02:21:31.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8abec19640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ab81411d0 con 0x7f8ab80a8bf0 2024-04-26T02:21:31.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8abec19640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ab8142990 con 0x7f8ab8141b30 2024-04-26T02:21:31.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab7fff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f8ab813f9e0 0x7f8ab8141420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53598/0 (socket says 172.21.15.87:53598) 2024-04-26T02:21:31.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab7fff640 1 -- 172.21.15.87:0/106427126 learned_addr learned my addr 172.21.15.87:0/106427126 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:31.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1149001859 0 0) 0x7f8ab8144ce0 con 0x7f8ab813f9e0 2024-04-26T02:21:31.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8aa4003530 con 0x7f8ab813f9e0 2024-04-26T02:21:31.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2808395881 0 0) 0x7f8ab8142990 con 0x7f8ab8141b30 2024-04-26T02:21:31.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ab8144ce0 con 0x7f8ab8141b30 2024-04-26T02:21:31.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2547495221 0 0) 0x7f8aa4003530 con 0x7f8ab813f9e0 2024-04-26T02:21:31.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ab8142990 con 0x7f8ab813f9e0 2024-04-26T02:21:31.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8aac001be0 con 0x7f8ab813f9e0 2024-04-26T02:21:31.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3264361699 0 0) 0x7f8ab8144ce0 con 0x7f8ab8141b30 2024-04-26T02:21:31.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8aa4003530 con 0x7f8ab8141b30 2024-04-26T02:21:31.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8ab0001ae0 con 0x7f8ab8141b30 2024-04-26T02:21:31.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4140600555 0 0) 0x7f8ab81411d0 con 0x7f8ab80a8bf0 2024-04-26T02:21:31.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ab8144ce0 con 0x7f8ab80a8bf0 2024-04-26T02:21:31.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3489344428 0 0) 0x7f8ab8142990 con 0x7f8ab813f9e0 2024-04-26T02:21:31.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 >> v1:172.21.15.87:6790/0 conn(0x7f8ab8141b30 legacy=0x7f8ab81e1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:31.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 >> v1:172.21.15.122:6789/0 conn(0x7f8ab80a8bf0 legacy=0x7f8ab8140ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:31.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ab8142c70 con 0x7f8ab813f9e0 2024-04-26T02:21:31.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8abec19640 1 -- 172.21.15.87:0/106427126 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8ab81368d0 con 0x7f8ab813f9e0 2024-04-26T02:21:31.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8abec19640 1 -- 172.21.15.87:0/106427126 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8ab81e3690 con 0x7f8ab813f9e0 2024-04-26T02:21:31.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8aac004cd0 con 0x7f8ab813f9e0 2024-04-26T02:21:31.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.569+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8aac005020 con 0x7f8ab813f9e0 2024-04-26T02:21:31.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.573+0000 7f8abec19640 1 -- 172.21.15.87:0/106427126 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a98005090 con 0x7f8ab813f9e0 2024-04-26T02:21:31.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.573+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) ==== 87354+0+0 (unknown 4190153105 0 0) 0x7f8aac005240 con 0x7f8ab813f9e0 2024-04-26T02:21:31.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.573+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 365644488 0 0) 0x7f8aac081220 con 0x7f8ab813f9e0 2024-04-26T02:21:31.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.577+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f8aac052460 con 0x7f8ab813f9e0 2024-04-26T02:21:31.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.777+0000 7f8abec19640 1 -- 172.21.15.87:0/106427126 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f8a98005380 con 0x7f8ab813f9e0 2024-04-26T02:21:31.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.781+0000 7f8ab57fa640 1 -- 172.21.15.87:0/106427126 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) ==== 74+0+166737 (unknown 3797009233 0 4250603589) 0x7f8aac051d60 con 0x7f8ab813f9e0 2024-04-26T02:21:31.783 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:21:31.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.785+0000 7f8abec19640 1 -- 172.21.15.87:0/106427126 >> v1:172.21.15.122:6816/416193625 conn(0x7f8aa4068350 legacy=0x7f8aa406a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:31.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.785+0000 7f8abec19640 1 -- 172.21.15.87:0/106427126 >> v1:172.21.15.87:6789/0 conn(0x7f8ab813f9e0 legacy=0x7f8ab8141420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:31.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.785+0000 7f8abec19640 1 -- 172.21.15.87:0/106427126 shutdown_connections 2024-04-26T02:21:31.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.785+0000 7f8abec19640 1 -- 172.21.15.87:0/106427126 >> 172.21.15.87:0/106427126 conn(0x7f8ab80b7af0 msgr2=0x7f8ab80b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:31.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.785+0000 7f8abec19640 1 -- 172.21.15.87:0/106427126 shutdown_connections 2024-04-26T02:21:31.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.785+0000 7f8abec19640 1 -- 172.21.15.87:0/106427126 wait complete. 2024-04-26T02:21:31.800 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":4,"flags":0,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6816","nonce":416193625}]},"active_addr":"172.21.15.122:6816/416193625","active_change":"2024-04-26T02:21:23.328676+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":3251946697}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":2363441929}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":2052691133}]}]} 2024-04-26T02:21:31.806 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-26T02:21:31.806 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-26T02:21:31.806 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T02:21:31.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.921+0000 7f074b3e6640 1 Processor -- start 2024-04-26T02:21:31.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.921+0000 7f074b3e6640 1 -- start start 2024-04-26T02:21:31.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f074b3e6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0744071f40 con 0x7f074406fe70 2024-04-26T02:21:31.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f074b3e6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f074413be10 con 0x7f074406fa90 2024-04-26T02:21:31.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f074b3e6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0744070e00 con 0x7f074413e510 2024-04-26T02:21:31.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f0749983640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f074413e510 0x7f0744149e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40198/0 (socket says 172.21.15.87:40198) 2024-04-26T02:21:31.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f0749983640 1 -- 172.21.15.87:0/3817251878 learned_addr learned my addr 172.21.15.87:0/3817251878 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:31.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f0748981640 1 --1- 172.21.15.87:0/3817251878 >> v1:172.21.15.87:6789/0 conn(0x7f074406fe70 0x7f074413b700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53614/0 (socket says 172.21.15.87:53614) 2024-04-26T02:21:31.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f073bfff640 1 -- 172.21.15.87:0/3817251878 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 565145548 0 0) 0x7f0744070e00 con 0x7f074413e510 2024-04-26T02:21:31.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f073bfff640 1 -- 172.21.15.87:0/3817251878 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0730003530 con 0x7f074413e510 2024-04-26T02:21:31.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f073bfff640 1 -- 172.21.15.87:0/3817251878 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 79854541 0 0) 0x7f0744071f40 con 0x7f074406fe70 2024-04-26T02:21:31.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f073bfff640 1 -- 172.21.15.87:0/3817251878 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0744070e00 con 0x7f074406fe70 2024-04-26T02:21:31.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f073bfff640 1 -- 172.21.15.87:0/3817251878 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3063384460 0 0) 0x7f0730003530 con 0x7f074413e510 2024-04-26T02:21:31.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f073bfff640 1 -- 172.21.15.87:0/3817251878 >> v1:172.21.15.122:6789/0 conn(0x7f074406fa90 legacy=0x7f0744072740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:31.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f073bfff640 1 -- 172.21.15.87:0/3817251878 >> v1:172.21.15.87:6789/0 conn(0x7f074406fe70 legacy=0x7f074413b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:31.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f073bfff640 1 -- 172.21.15.87:0/3817251878 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0744070a40 con 0x7f074413e510 2024-04-26T02:21:31.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f073bfff640 1 -- 172.21.15.87:0/3817251878 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f073c0019a0 con 0x7f074413e510 2024-04-26T02:21:31.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f073bfff640 1 -- 172.21.15.87:0/3817251878 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f073c001cc0 con 0x7f074413e510 2024-04-26T02:21:31.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f073bfff640 1 -- 172.21.15.87:0/3817251878 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f073c0020d0 con 0x7f074413e510 2024-04-26T02:21:31.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f074b3e6640 1 -- 172.21.15.87:0/3817251878 >> v1:172.21.15.87:6790/0 conn(0x7f074413e510 legacy=0x7f0744149e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:31.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f074b3e6640 1 -- 172.21.15.87:0/3817251878 shutdown_connections 2024-04-26T02:21:31.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f074b3e6640 1 -- 172.21.15.87:0/3817251878 >> 172.21.15.87:0/3817251878 conn(0x7f07440b4230 msgr2=0x7f074406b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:31.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f074b3e6640 1 -- 172.21.15.87:0/3817251878 shutdown_connections 2024-04-26T02:21:31.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f074b3e6640 1 -- 172.21.15.87:0/3817251878 wait complete. 2024-04-26T02:21:31.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f074b3e6640 1 Processor -- start 2024-04-26T02:21:31.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.925+0000 7f074b3e6640 1 -- start start 2024-04-26T02:21:31.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f074b3e6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07440aadc0 con 0x7f074413e510 2024-04-26T02:21:31.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f074b3e6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07440ab030 con 0x7f074406fe70 2024-04-26T02:21:31.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f074b3e6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07440ab310 con 0x7f074406fa90 2024-04-26T02:21:31.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0749182640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f074406fa90 0x7f07440a8f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40208/0 (socket says 172.21.15.87:40208) 2024-04-26T02:21:31.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0749182640 1 -- 172.21.15.87:0/3343559578 learned_addr learned my addr 172.21.15.87:0/3343559578 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:31.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 572185708 0 0) 0x7f07440ab310 con 0x7f074406fa90 2024-04-26T02:21:31.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0728003530 con 0x7f074406fa90 2024-04-26T02:21:31.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2215003621 0 0) 0x7f07440aadc0 con 0x7f074413e510 2024-04-26T02:21:31.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07440ab310 con 0x7f074413e510 2024-04-26T02:21:31.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3904095442 0 0) 0x7f0728003530 con 0x7f074406fa90 2024-04-26T02:21:31.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07440aadc0 con 0x7f074406fa90 2024-04-26T02:21:31.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f0740001920 con 0x7f074406fa90 2024-04-26T02:21:31.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2682446 0 0) 0x7f07440ab310 con 0x7f074413e510 2024-04-26T02:21:31.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0728003530 con 0x7f074413e510 2024-04-26T02:21:31.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f073c001960 con 0x7f074413e510 2024-04-26T02:21:31.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1515034136 0 0) 0x7f07440ab030 con 0x7f074406fe70 2024-04-26T02:21:31.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07440ab310 con 0x7f074406fe70 2024-04-26T02:21:31.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1329999914 0 0) 0x7f07440aadc0 con 0x7f074406fa90 2024-04-26T02:21:31.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 >> v1:172.21.15.122:6789/0 conn(0x7f074406fe70 legacy=0x7f07441408e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:31.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 >> v1:172.21.15.87:6789/0 conn(0x7f074413e510 legacy=0x7f07440acc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:31.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07440a9af0 con 0x7f074406fa90 2024-04-26T02:21:31.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f074b3e6640 1 -- 172.21.15.87:0/3343559578 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f07440a9780 con 0x7f074406fa90 2024-04-26T02:21:31.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f074b3e6640 1 -- 172.21.15.87:0/3343559578 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f07440aa1e0 con 0x7f074406fa90 2024-04-26T02:21:31.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0740001d00 con 0x7f074406fa90 2024-04-26T02:21:31.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f0740003af0 con 0x7f074406fa90 2024-04-26T02:21:31.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f074b3e6640 1 -- 172.21.15.87:0/3343559578 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0724005090 con 0x7f074406fa90 2024-04-26T02:21:31.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) ==== 87354+0+0 (unknown 4190153105 0 0) 0x7f0740003c90 con 0x7f074406fa90 2024-04-26T02:21:31.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.929+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 365644488 0 0) 0x7f0740080fe0 con 0x7f074406fa90 2024-04-26T02:21:31.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:31.933+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f07440aa1e0 con 0x7f074406fa90 2024-04-26T02:21:32.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.093+0000 7f074b3e6640 1 -- 172.21.15.87:0/3343559578 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0724005380 con 0x7f074406fa90 2024-04-26T02:21:32.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.097+0000 7f0739ffb640 1 -- 172.21.15.87:0/3343559578 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12727 (unknown 3225238715 0 3939590) 0x7f07400520e0 con 0x7f074406fa90 2024-04-26T02:21:32.095 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:21:32.095 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":17,"fsid":"6232cbab-48ed-4660-9de8-286f4c0e9a32","created":"2024-04-26T02:21:18.617390+0000","modified":"2024-04-26T02:21:31.402919+0000","last_up_change":"2024-04-26T02:21:28.375870+0000","last_in_change":"2024-04-26T02:21:20.860445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T02:21:27.425849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T02:21:29.621425+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6ea27580-6f9b-4e32-813f-fb8923bb6180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6804","nonce":815800445}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6805","nonce":815800445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6807","nonce":815800445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6806","nonce":815800445}]},"public_addr":"172.21.15.87:6804/815800445","cluster_addr":"172.21.15.87:6805/815800445","heartbeat_back_addr":"172.21.15.87:6807/815800445","heartbeat_front_addr":"172.21.15.87:6806/815800445","state":["exists","up"]},{"osd":1,"uuid":"b8afe0d9-5e03-4ae1-b91d-54b1b7071d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6800","nonce":174846219}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6801","nonce":174846219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6803","nonce":174846219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6802","nonce":174846219}]},"public_addr":"172.21.15.87:6800/174846219","cluster_addr":"172.21.15.87:6801/174846219","heartbeat_back_addr":"172.21.15.87:6803/174846219","heartbeat_front_addr":"172.21.15.87:6802/174846219","state":["exists","up"]},{"osd":2,"uuid":"d3a9234e-ef70-421d-b5ba-121f4f5cdcc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6808","nonce":2992851618}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6809","nonce":2992851618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6811","nonce":2992851618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6810","nonce":2992851618}]},"public_addr":"172.21.15.87:6808/2992851618","cluster_addr":"172.21.15.87:6809/2992851618","heartbeat_back_addr":"172.21.15.87:6811/2992851618","heartbeat_front_addr":"172.21.15.87:6810/2992851618","state":["exists","up"]},{"osd":3,"uuid":"bfc148fb-9fd7-4a94-8287-52a4f38d7100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6812","nonce":3926229128}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6813","nonce":3926229128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6815","nonce":3926229128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6814","nonce":3926229128}]},"public_addr":"172.21.15.87:6812/3926229128","cluster_addr":"172.21.15.87:6813/3926229128","heartbeat_back_addr":"172.21.15.87:6815/3926229128","heartbeat_front_addr":"172.21.15.87:6814/3926229128","state":["exists","up"]},{"osd":4,"uuid":"3af86762-d77b-4c24-ac59-d0f20c4d423c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6800","nonce":850918874}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6801","nonce":850918874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6803","nonce":850918874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6802","nonce":850918874}]},"public_addr":"172.21.15.122:6800/850918874","cluster_addr":"172.21.15.122:6801/850918874","heartbeat_back_addr":"172.21.15.122:6803/850918874","heartbeat_front_addr":"172.21.15.122:6802/850918874","state":["exists","up"]},{"osd":5,"uuid":"6a28bfec-7754-4a56-bbd1-0d1d2655fde7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6812","nonce":498384109}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6813","nonce":498384109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6815","nonce":498384109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6814","nonce":498384109}]},"public_addr":"172.21.15.122:6812/498384109","cluster_addr":"172.21.15.122:6813/498384109","heartbeat_back_addr":"172.21.15.122:6815/498384109","heartbeat_front_addr":"172.21.15.122:6814/498384109","state":["exists","up"]},{"osd":6,"uuid":"db087c2e-bd80-4863-a1db-5fbd8f9fe083","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6805","nonce":662791991}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6807","nonce":662791991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6811","nonce":662791991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6810","nonce":662791991}]},"public_addr":"172.21.15.122:6805/662791991","cluster_addr":"172.21.15.122:6807/662791991","heartbeat_back_addr":"172.21.15.122:6811/662791991","heartbeat_front_addr":"172.21.15.122:6810/662791991","state":["exists","up"]},{"osd":7,"uuid":"ccf8f7a9-c44f-44f4-91d1-9f197803f67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6804","nonce":7687745}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6806","nonce":7687745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6809","nonce":7687745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6808","nonce":7687745}]},"public_addr":"172.21.15.122:6804/7687745","cluster_addr":"172.21.15.122:6806/7687745","heartbeat_back_addr":"172.21.15.122:6809/7687745","heartbeat_front_addr":"172.21.15.122:6808/7687745","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.424114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.466699+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.335478+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.391653+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.462086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.410699+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:25.564818+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.378766+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:21:32.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.097+0000 7f074b3e6640 1 -- 172.21.15.87:0/3343559578 >> v1:172.21.15.122:6816/416193625 conn(0x7f07280683c0 legacy=0x7f072806a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.097+0000 7f074b3e6640 1 -- 172.21.15.87:0/3343559578 >> v1:172.21.15.87:6790/0 conn(0x7f074406fa90 legacy=0x7f07440a8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.097+0000 7f074b3e6640 1 -- 172.21.15.87:0/3343559578 shutdown_connections 2024-04-26T02:21:32.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.097+0000 7f074b3e6640 1 -- 172.21.15.87:0/3343559578 >> 172.21.15.87:0/3343559578 conn(0x7f07440b4230 msgr2=0x7f0744074c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.097+0000 7f074b3e6640 1 -- 172.21.15.87:0/3343559578 shutdown_connections 2024-04-26T02:21:32.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.097+0000 7f074b3e6640 1 -- 172.21.15.87:0/3343559578 wait complete. 2024-04-26T02:21:32.114 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-26T02:21:32.114 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T02:21:32.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.233+0000 7f2923ba0640 1 Processor -- start 2024-04-26T02:21:32.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.233+0000 7f2923ba0640 1 -- start start 2024-04-26T02:21:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f2923ba0640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291c0709a0 con 0x7f291c13f590 2024-04-26T02:21:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f2923ba0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291c072210 con 0x7f291c0ae580 2024-04-26T02:21:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f2923ba0640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291c0724f0 con 0x7f291c13c820 2024-04-26T02:21:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f292113b640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f291c13c820 0x7f291c13ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40212/0 (socket says 172.21.15.87:40212) 2024-04-26T02:21:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f292113b640 1 -- 172.21.15.87:0/202877304 learned_addr learned my addr 172.21.15.87:0/202877304 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f292093a640 1 -- 172.21.15.87:0/202877304 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1292699101 0 0) 0x7f291c0724f0 con 0x7f291c13c820 2024-04-26T02:21:32.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f292093a640 1 -- 172.21.15.87:0/202877304 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2908005740 con 0x7f291c13c820 2024-04-26T02:21:32.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f292093a640 1 -- 172.21.15.87:0/202877304 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3103249261 0 0) 0x7f291c0709a0 con 0x7f291c13f590 2024-04-26T02:21:32.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f292093a640 1 -- 172.21.15.87:0/202877304 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f291c0724f0 con 0x7f291c13f590 2024-04-26T02:21:32.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f292093a640 1 -- 172.21.15.87:0/202877304 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4266687544 0 0) 0x7f291c0724f0 con 0x7f291c13f590 2024-04-26T02:21:32.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f292093a640 1 -- 172.21.15.87:0/202877304 >> v1:172.21.15.87:6790/0 conn(0x7f291c13c820 legacy=0x7f291c13ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f292093a640 1 -- 172.21.15.87:0/202877304 >> v1:172.21.15.122:6789/0 conn(0x7f291c0ae580 legacy=0x7f291c13c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f292093a640 1 -- 172.21.15.87:0/202877304 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291c0727d0 con 0x7f291c13f590 2024-04-26T02:21:32.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f292093a640 1 -- 172.21.15.87:0/202877304 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f29140019a0 con 0x7f291c13f590 2024-04-26T02:21:32.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f292093a640 1 -- 172.21.15.87:0/202877304 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2914001b90 con 0x7f291c13f590 2024-04-26T02:21:32.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f292093a640 1 -- 172.21.15.87:0/202877304 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f2914001fa0 con 0x7f291c13f590 2024-04-26T02:21:32.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f2923ba0640 1 -- 172.21.15.87:0/202877304 >> v1:172.21.15.87:6789/0 conn(0x7f291c13f590 legacy=0x7f291c14a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f2923ba0640 1 -- 172.21.15.87:0/202877304 shutdown_connections 2024-04-26T02:21:32.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f2923ba0640 1 -- 172.21.15.87:0/202877304 >> 172.21.15.87:0/202877304 conn(0x7f291c0b4230 msgr2=0x7f291c06b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f2923ba0640 1 -- 172.21.15.87:0/202877304 shutdown_connections 2024-04-26T02:21:32.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f2923ba0640 1 -- 172.21.15.87:0/202877304 wait complete. 2024-04-26T02:21:32.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f2923ba0640 1 Processor -- start 2024-04-26T02:21:32.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.237+0000 7f2923ba0640 1 -- start start 2024-04-26T02:21:32.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f2923ba0640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291c142dd0 con 0x7f291c0ae580 2024-04-26T02:21:32.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f2923ba0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291c1d0790 con 0x7f291c13f590 2024-04-26T02:21:32.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f2923ba0640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291c143180 con 0x7f291c13c820 2024-04-26T02:21:32.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f292193c640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f291c0ae580 0x7f291c1414c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53638/0 (socket says 172.21.15.87:53638) 2024-04-26T02:21:32.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f292193c640 1 -- 172.21.15.87:0/2456225356 learned_addr learned my addr 172.21.15.87:0/2456225356 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1799039826 0 0) 0x7f291c143180 con 0x7f291c13c820 2024-04-26T02:21:32.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2900003530 con 0x7f291c13c820 2024-04-26T02:21:32.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3899926232 0 0) 0x7f291c142dd0 con 0x7f291c0ae580 2024-04-26T02:21:32.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f291c143180 con 0x7f291c0ae580 2024-04-26T02:21:32.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3194013058 0 0) 0x7f2900003530 con 0x7f291c13c820 2024-04-26T02:21:32.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f291c142dd0 con 0x7f291c13c820 2024-04-26T02:21:32.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f290c001320 con 0x7f291c13c820 2024-04-26T02:21:32.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1009536651 0 0) 0x7f291c143180 con 0x7f291c0ae580 2024-04-26T02:21:32.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2900003530 con 0x7f291c0ae580 2024-04-26T02:21:32.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f29180018c0 con 0x7f291c0ae580 2024-04-26T02:21:32.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 990277021 0 0) 0x7f291c1d0790 con 0x7f291c13f590 2024-04-26T02:21:32.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f291c143180 con 0x7f291c13f590 2024-04-26T02:21:32.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3409471597 0 0) 0x7f291c142dd0 con 0x7f291c13c820 2024-04-26T02:21:32.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 >> v1:172.21.15.122:6789/0 conn(0x7f291c13f590 legacy=0x7f291c1d2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 >> v1:172.21.15.87:6789/0 conn(0x7f291c0ae580 legacy=0x7f291c1414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291c141bd0 con 0x7f291c13c820 2024-04-26T02:21:32.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f2923ba0640 1 -- 172.21.15.87:0/2456225356 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f291c1340e0 con 0x7f291c13c820 2024-04-26T02:21:32.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f2923ba0640 1 -- 172.21.15.87:0/2456225356 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f291c1d3440 con 0x7f291c13c820 2024-04-26T02:21:32.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f290c0047f0 con 0x7f291c13c820 2024-04-26T02:21:32.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f290c004b40 con 0x7f291c13c820 2024-04-26T02:21:32.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f2923ba0640 1 -- 172.21.15.87:0/2456225356 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2904005090 con 0x7f291c13c820 2024-04-26T02:21:32.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) ==== 87354+0+0 (unknown 4190153105 0 0) 0x7f290c01a230 con 0x7f291c13c820 2024-04-26T02:21:32.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.241+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(17..17 src has 1..17) ==== 4090+0+0 (unknown 365644488 0 0) 0x7f290c080f60 con 0x7f291c13c820 2024-04-26T02:21:32.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.245+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f290c052060 con 0x7f291c13c820 2024-04-26T02:21:32.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.405+0000 7f2923ba0640 1 -- 172.21.15.87:0/2456225356 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2904005380 con 0x7f291c13c820 2024-04-26T02:21:32.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.405+0000 7f29127fc640 1 -- 172.21.15.87:0/2456225356 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12727 (unknown 3225238715 0 3939590) 0x7f290c051960 con 0x7f291c13c820 2024-04-26T02:21:32.404 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:21:32.405 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":17,"fsid":"6232cbab-48ed-4660-9de8-286f4c0e9a32","created":"2024-04-26T02:21:18.617390+0000","modified":"2024-04-26T02:21:31.402919+0000","last_up_change":"2024-04-26T02:21:28.375870+0000","last_in_change":"2024-04-26T02:21:20.860445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T02:21:27.425849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T02:21:29.621425+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6ea27580-6f9b-4e32-813f-fb8923bb6180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6804","nonce":815800445}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6805","nonce":815800445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6807","nonce":815800445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6806","nonce":815800445}]},"public_addr":"172.21.15.87:6804/815800445","cluster_addr":"172.21.15.87:6805/815800445","heartbeat_back_addr":"172.21.15.87:6807/815800445","heartbeat_front_addr":"172.21.15.87:6806/815800445","state":["exists","up"]},{"osd":1,"uuid":"b8afe0d9-5e03-4ae1-b91d-54b1b7071d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6800","nonce":174846219}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6801","nonce":174846219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6803","nonce":174846219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6802","nonce":174846219}]},"public_addr":"172.21.15.87:6800/174846219","cluster_addr":"172.21.15.87:6801/174846219","heartbeat_back_addr":"172.21.15.87:6803/174846219","heartbeat_front_addr":"172.21.15.87:6802/174846219","state":["exists","up"]},{"osd":2,"uuid":"d3a9234e-ef70-421d-b5ba-121f4f5cdcc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6808","nonce":2992851618}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6809","nonce":2992851618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6811","nonce":2992851618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6810","nonce":2992851618}]},"public_addr":"172.21.15.87:6808/2992851618","cluster_addr":"172.21.15.87:6809/2992851618","heartbeat_back_addr":"172.21.15.87:6811/2992851618","heartbeat_front_addr":"172.21.15.87:6810/2992851618","state":["exists","up"]},{"osd":3,"uuid":"bfc148fb-9fd7-4a94-8287-52a4f38d7100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6812","nonce":3926229128}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6813","nonce":3926229128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6815","nonce":3926229128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6814","nonce":3926229128}]},"public_addr":"172.21.15.87:6812/3926229128","cluster_addr":"172.21.15.87:6813/3926229128","heartbeat_back_addr":"172.21.15.87:6815/3926229128","heartbeat_front_addr":"172.21.15.87:6814/3926229128","state":["exists","up"]},{"osd":4,"uuid":"3af86762-d77b-4c24-ac59-d0f20c4d423c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6800","nonce":850918874}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6801","nonce":850918874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6803","nonce":850918874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6802","nonce":850918874}]},"public_addr":"172.21.15.122:6800/850918874","cluster_addr":"172.21.15.122:6801/850918874","heartbeat_back_addr":"172.21.15.122:6803/850918874","heartbeat_front_addr":"172.21.15.122:6802/850918874","state":["exists","up"]},{"osd":5,"uuid":"6a28bfec-7754-4a56-bbd1-0d1d2655fde7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6812","nonce":498384109}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6813","nonce":498384109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6815","nonce":498384109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6814","nonce":498384109}]},"public_addr":"172.21.15.122:6812/498384109","cluster_addr":"172.21.15.122:6813/498384109","heartbeat_back_addr":"172.21.15.122:6815/498384109","heartbeat_front_addr":"172.21.15.122:6814/498384109","state":["exists","up"]},{"osd":6,"uuid":"db087c2e-bd80-4863-a1db-5fbd8f9fe083","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6805","nonce":662791991}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6807","nonce":662791991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6811","nonce":662791991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6810","nonce":662791991}]},"public_addr":"172.21.15.122:6805/662791991","cluster_addr":"172.21.15.122:6807/662791991","heartbeat_back_addr":"172.21.15.122:6811/662791991","heartbeat_front_addr":"172.21.15.122:6810/662791991","state":["exists","up"]},{"osd":7,"uuid":"ccf8f7a9-c44f-44f4-91d1-9f197803f67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6804","nonce":7687745}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6806","nonce":7687745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6809","nonce":7687745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6808","nonce":7687745}]},"public_addr":"172.21.15.122:6804/7687745","cluster_addr":"172.21.15.122:6806/7687745","heartbeat_back_addr":"172.21.15.122:6809/7687745","heartbeat_front_addr":"172.21.15.122:6808/7687745","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.424114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.466699+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.335478+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.391653+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.462086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.410699+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:25.564818+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.378766+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:21:32.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.409+0000 7f2923ba0640 1 -- 172.21.15.87:0/2456225356 >> v1:172.21.15.122:6816/416193625 conn(0x7f2900068380 legacy=0x7f290006a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.409+0000 7f2923ba0640 1 -- 172.21.15.87:0/2456225356 >> v1:172.21.15.87:6790/0 conn(0x7f291c13c820 legacy=0x7f291c1339d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.409+0000 7f2923ba0640 1 -- 172.21.15.87:0/2456225356 shutdown_connections 2024-04-26T02:21:32.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.409+0000 7f2923ba0640 1 -- 172.21.15.87:0/2456225356 >> 172.21.15.87:0/2456225356 conn(0x7f291c0b4230 msgr2=0x7f291c06b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.409+0000 7f2923ba0640 1 -- 172.21.15.87:0/2456225356 shutdown_connections 2024-04-26T02:21:32.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.409+0000 7f2923ba0640 1 -- 172.21.15.87:0/2456225356 wait complete. 2024-04-26T02:21:32.424 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-26T02:21:32.425 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-26T02:21:32.425 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-26T02:21:32.426 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-26T02:21:32.426 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-26T02:21:32.426 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-26T02:21:32.427 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-26T02:21:32.427 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-26T02:21:32.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc7b759e640 1 Processor -- start 2024-04-26T02:21:32.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc7b759e640 1 -- start start 2024-04-26T02:21:32.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f32a747f640 1 Processor -- start 2024-04-26T02:21:32.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f32a747f640 1 -- start start 2024-04-26T02:21:32.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc7b759e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7b002d550 con 0x7fc7b0037160 2024-04-26T02:21:32.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc7b759e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7b002b2f0 con 0x7fc7b002ad40 2024-04-26T02:21:32.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc7b759e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7b00c1d50 con 0x7fc7b002c210 2024-04-26T02:21:32.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9db1946640 1 Processor -- start 2024-04-26T02:21:32.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc7b5d9b640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fc7b002c210 0x7fc7b00c7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40244/0 (socket says 172.21.15.87:40244) 2024-04-26T02:21:32.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc7b659c640 1 --1- 172.21.15.87:0/1230501657 >> v1:172.21.15.87:6789/0 conn(0x7fc7b0037160 0x7fc7b00c1640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53650/0 (socket says 172.21.15.87:53650) 2024-04-26T02:21:32.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc7b5d9b640 1 -- 172.21.15.87:0/1230501657 learned_addr learned my addr 172.21.15.87:0/1230501657 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f32a747f640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32a00adda0 con 0x7f32a0072a80 2024-04-26T02:21:32.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f32a747f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32a00aec50 con 0x7f32a006b590 2024-04-26T02:21:32.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f32a747f640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32a00ae080 con 0x7f32a006b1b0 2024-04-26T02:21:32.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9db1946640 1 -- start start 2024-04-26T02:21:32.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc7b5d9b640 1 -- 172.21.15.87:0/1230501657 >> v1:172.21.15.87:6790/0 conn(0x7fc7b002c210 legacy=0x7fc7b00c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T02:21:32.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc7b5d9b640 1 -- 172.21.15.87:0/1230501657 >> v1:172.21.15.87:6790/0 conn(0x7fc7b002c210 legacy=0x7fc7b00c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T02:21:32.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc7b5d9b640 1 --1- 172.21.15.87:0/1230501657 >> v1:172.21.15.87:6790/0 conn(0x7fc7b002c210 0x7fc7b00c7a10 :-1 s=OPENED pgs=83 cs=1 l=1).handle_message read tag failed 2024-04-26T02:21:32.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc7b5d9b640 1 --1- 172.21.15.87:0/1230501657 >> v1:172.21.15.87:6790/0 conn(0x7fc7b002c210 0x7fc7b00c7a10 :-1 s=OPENED pgs=83 cs=1 l=1).fault on lossy channel, failing 2024-04-26T02:21:32.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f32a521b640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f32a006b1b0 0x7f32a00b07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40256/0 (socket says 172.21.15.87:40256) 2024-04-26T02:21:32.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f32a521b640 1 -- 172.21.15.87:0/722668005 learned_addr learned my addr 172.21.15.87:0/722668005 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f3293fff640 1 -- 172.21.15.87:0/722668005 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3592744954 0 0) 0x7f32a00ae080 con 0x7f32a006b1b0 2024-04-26T02:21:32.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f3293fff640 1 -- 172.21.15.87:0/722668005 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f328c003530 con 0x7f32a006b1b0 2024-04-26T02:21:32.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f3293fff640 1 -- 172.21.15.87:0/722668005 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3190444976 0 0) 0x7f32a00aec50 con 0x7f32a006b590 2024-04-26T02:21:32.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f3293fff640 1 -- 172.21.15.87:0/722668005 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32a00ae080 con 0x7f32a006b590 2024-04-26T02:21:32.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f3293fff640 1 -- 172.21.15.87:0/722668005 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3827779134 0 0) 0x7f32a00adda0 con 0x7f32a0072a80 2024-04-26T02:21:32.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f3293fff640 1 -- 172.21.15.87:0/722668005 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32a00aec50 con 0x7f32a0072a80 2024-04-26T02:21:32.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc5cd81e640 1 Processor -- start 2024-04-26T02:21:32.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f3293fff640 1 -- 172.21.15.87:0/722668005 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3733786581 0 0) 0x7f32a00ae080 con 0x7f32a006b590 2024-04-26T02:21:32.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f3293fff640 1 -- 172.21.15.87:0/722668005 >> v1:172.21.15.87:6790/0 conn(0x7f32a006b1b0 legacy=0x7f32a00b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f3293fff640 1 -- 172.21.15.87:0/722668005 >> v1:172.21.15.87:6789/0 conn(0x7f32a0072a80 legacy=0x7f32a014da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f3293fff640 1 -- 172.21.15.87:0/722668005 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a00aceb0 con 0x7f32a006b590 2024-04-26T02:21:32.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f3293fff640 1 -- 172.21.15.87:0/722668005 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f329c001600 con 0x7f32a006b590 2024-04-26T02:21:32.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9db1946640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dac13fb40 con 0x7f9dac072750 2024-04-26T02:21:32.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9db1946640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dac06f980 con 0x7f9dac072370 2024-04-26T02:21:32.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9db1946640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dac06fc60 con 0x7f9dac0af2b0 2024-04-26T02:21:32.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9dabfff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f9dac072750 0x7f9dac141f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53662/0 (socket says 172.21.15.87:53662) 2024-04-26T02:21:32.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f5f03554640 1 Processor -- start 2024-04-26T02:21:32.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9dabfff640 1 -- 172.21.15.87:0/3739085866 learned_addr learned my addr 172.21.15.87:0/3739085866 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc5cd81e640 1 -- start start 2024-04-26T02:21:32.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f3293fff640 1 -- 172.21.15.87:0/722668005 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f329c001960 con 0x7f32a006b590 2024-04-26T02:21:32.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f3293fff640 1 -- 172.21.15.87:0/722668005 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f329c001d30 con 0x7f32a006b590 2024-04-26T02:21:32.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9dab7fe640 1 -- 172.21.15.87:0/3739085866 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3670851968 0 0) 0x7f9dac13fb40 con 0x7f9dac072750 2024-04-26T02:21:32.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9dab7fe640 1 -- 172.21.15.87:0/3739085866 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d9c003530 con 0x7f9dac072750 2024-04-26T02:21:32.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9dab7fe640 1 -- 172.21.15.87:0/3739085866 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 323940001 0 0) 0x7f9dac06fc60 con 0x7f9dac0af2b0 2024-04-26T02:21:32.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9dab7fe640 1 -- 172.21.15.87:0/3739085866 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dac13fb40 con 0x7f9dac0af2b0 2024-04-26T02:21:32.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9dab7fe640 1 -- 172.21.15.87:0/3739085866 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 645676533 0 0) 0x7f9d9c003530 con 0x7f9dac072750 2024-04-26T02:21:32.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9dab7fe640 1 -- 172.21.15.87:0/3739085866 >> v1:172.21.15.87:6790/0 conn(0x7f9dac0af2b0 legacy=0x7f9dac14d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9dab7fe640 1 -- 172.21.15.87:0/3739085866 >> v1:172.21.15.122:6789/0 conn(0x7f9dac072370 legacy=0x7f9dac13f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:21:32.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9dab7fe640 1 -- 172.21.15.87:0/3739085866 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dac0ac8e0 con 0x7f9dac072750 2024-04-26T02:21:32.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9dab7fe640 1 -- 172.21.15.87:0/3739085866 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9da0001620 con 0x7f9dac072750 2024-04-26T02:21:32.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f85514c3640 1 Processor -- start 2024-04-26T02:21:32.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f85514c3640 1 -- start start 2024-04-26T02:21:32.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f5f03554640 1 -- start start 2024-04-26T02:21:32.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc5cd81e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c8070e60 con 0x7fc5c80af380 2024-04-26T02:21:32.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc5cd81e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c80add00 con 0x7fc5c8143c10 2024-04-26T02:21:32.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc5c7fff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fc5c80703d0 0x7fc5c8072450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40272/0 (socket says 172.21.15.87:40272) 2024-04-26T02:21:32.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc5cd81e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c80afeb0 con 0x7fc5c80703d0 2024-04-26T02:21:32.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7fc5c7fff640 1 -- 172.21.15.87:0/756959603 learned_addr learned my addr 172.21.15.87:0/756959603 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f5f03554640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef402e330 con 0x7f5ef4036a60 2024-04-26T02:21:32.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f5f03554640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef402e610 con 0x7f5ef40c3080 2024-04-26T02:21:32.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f5f03554640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef402ce80 con 0x7f5ef40352a0 2024-04-26T02:21:32.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9dab7fe640 1 -- 172.21.15.87:0/3739085866 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9da0001880 con 0x7f9dac072750 2024-04-26T02:21:32.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f9dab7fe640 1 -- 172.21.15.87:0/3739085866 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9da0001c10 con 0x7f9dac072750 2024-04-26T02:21:32.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f5f012f0640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f5ef4036a60 0x7f5ef40c2970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53692/0 (socket says 172.21.15.87:53692) 2024-04-26T02:21:32.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.605+0000 7f5f012f0640 1 -- 172.21.15.87:0/520949763 learned_addr learned my addr 172.21.15.87:0/520949763 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330359e640 1 Processor -- start 2024-04-26T02:21:32.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5c77fe640 1 -- 172.21.15.87:0/756959603 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 836255550 0 0) 0x7fc5c80add00 con 0x7fc5c8143c10 2024-04-26T02:21:32.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5c77fe640 1 -- 172.21.15.87:0/756959603 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5b8003530 con 0x7fc5c8143c10 2024-04-26T02:21:32.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5c77fe640 1 -- 172.21.15.87:0/756959603 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3096436769 0 0) 0x7fc5c8070e60 con 0x7fc5c80af380 2024-04-26T02:21:32.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5c77fe640 1 -- 172.21.15.87:0/756959603 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5c80add00 con 0x7fc5c80af380 2024-04-26T02:21:32.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5c77fe640 1 -- 172.21.15.87:0/756959603 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4250077103 0 0) 0x7fc5c80afeb0 con 0x7fc5c80703d0 2024-04-26T02:21:32.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5c77fe640 1 -- 172.21.15.87:0/756959603 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5c8070e60 con 0x7fc5c80703d0 2024-04-26T02:21:32.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f85514c3640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f854c072170 con 0x7f854c13e9c0 2024-04-26T02:21:32.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f85514c3640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f854c072450 con 0x7f854c072bd0 2024-04-26T02:21:32.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f85514c3640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f854c071110 con 0x7f854c138460 2024-04-26T02:21:32.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f854affd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f854c072bd0 0x7f854c13e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.87:35436/0 (socket says 172.21.15.87:35436) 2024-04-26T02:21:32.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f854affd640 1 -- 172.21.15.87:0/3890468491 learned_addr learned my addr 172.21.15.87:0/3890468491 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f9db1946640 1 -- 172.21.15.87:0/3739085866 >> v1:172.21.15.87:6789/0 conn(0x7f9dac072750 legacy=0x7f9dac141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f32a747f640 1 -- 172.21.15.87:0/722668005 >> v1:172.21.15.122:6789/0 conn(0x7f32a006b590 legacy=0x7f32a0142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f32a747f640 1 -- 172.21.15.87:0/722668005 shutdown_connections 2024-04-26T02:21:32.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f32a747f640 1 -- 172.21.15.87:0/722668005 >> 172.21.15.87:0/722668005 conn(0x7f32a00b4a10 msgr2=0x7f32a00b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f9db1946640 1 -- 172.21.15.87:0/3739085866 shutdown_connections 2024-04-26T02:21:32.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f9db1946640 1 -- 172.21.15.87:0/3739085866 >> 172.21.15.87:0/3739085866 conn(0x7f9dac06b8c0 msgr2=0x7f9dac0b3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5c77fe640 1 -- 172.21.15.87:0/756959603 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3400904577 0 0) 0x7fc5c8070e60 con 0x7fc5c80703d0 2024-04-26T02:21:32.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5c77fe640 1 -- 172.21.15.87:0/756959603 >> v1:172.21.15.122:6789/0 conn(0x7fc5c8143c10 legacy=0x7fc5c8151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f32a747f640 1 -- 172.21.15.87:0/722668005 shutdown_connections 2024-04-26T02:21:32.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f32a747f640 1 -- 172.21.15.87:0/722668005 wait complete. 2024-04-26T02:21:32.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5c77fe640 1 -- 172.21.15.87:0/756959603 >> v1:172.21.15.87:6789/0 conn(0x7fc5c80af380 legacy=0x7fc5c8143500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5c77fe640 1 -- 172.21.15.87:0/756959603 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c8072b60 con 0x7fc5c80703d0 2024-04-26T02:21:32.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f9db1946640 1 -- 172.21.15.87:0/3739085866 shutdown_connections 2024-04-26T02:21:32.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330359e640 1 -- start start 2024-04-26T02:21:32.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5c77fe640 1 -- 172.21.15.87:0/756959603 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc5bc0015c0 con 0x7fc5c80703d0 2024-04-26T02:21:32.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330359e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32fc02e640 con 0x7f32fc036a00 2024-04-26T02:21:32.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330359e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32fc02d1b0 con 0x7f32fc036de0 2024-04-26T02:21:32.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330359e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32fc02ce80 con 0x7f32fc035240 2024-04-26T02:21:32.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5c77fe640 1 -- 172.21.15.87:0/756959603 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc5bc001820 con 0x7fc5c80703d0 2024-04-26T02:21:32.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f3301d9b640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f32fc035240 0x7f32fc0c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40286/0 (socket says 172.21.15.87:40286) 2024-04-26T02:21:32.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f3301d9b640 1 -- 172.21.15.87:0/596276965 learned_addr learned my addr 172.21.15.87:0/596276965 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc7b559a640 1 -- 172.21.15.87:0/1230501657 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2414326516 0 0) 0x7fc7b002b2f0 con 0x7fc7b002ad40 2024-04-26T02:21:32.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc7b559a640 1 -- 172.21.15.87:0/1230501657 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7a0003530 con 0x7fc7b002ad40 2024-04-26T02:21:32.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f5ef3fff640 1 -- 172.21.15.87:0/520949763 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3533084439 0 0) 0x7f5ef402e330 con 0x7f5ef4036a60 2024-04-26T02:21:32.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f5ef3fff640 1 -- 172.21.15.87:0/520949763 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ee8003530 con 0x7f5ef4036a60 2024-04-26T02:21:32.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f5ef3fff640 1 -- 172.21.15.87:0/520949763 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3183233861 0 0) 0x7f5ef402e610 con 0x7f5ef40c3080 2024-04-26T02:21:32.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f5ef3fff640 1 -- 172.21.15.87:0/520949763 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ef402e330 con 0x7f5ef40c3080 2024-04-26T02:21:32.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc7b559a640 1 -- 172.21.15.87:0/1230501657 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3087482075 0 0) 0x7fc7b002d550 con 0x7fc7b0037160 2024-04-26T02:21:32.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f8549ffb640 1 -- 172.21.15.87:0/3890468491 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2984075389 0 0) 0x7f854c072450 con 0x7f854c072bd0 2024-04-26T02:21:32.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f8549ffb640 1 -- 172.21.15.87:0/3890468491 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8534003530 con 0x7f854c072bd0 2024-04-26T02:21:32.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f8549ffb640 1 -- 172.21.15.87:0/3890468491 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 763268040 0 0) 0x7f854c072170 con 0x7f854c13e9c0 2024-04-26T02:21:32.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f8549ffb640 1 -- 172.21.15.87:0/3890468491 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f854c072450 con 0x7f854c13e9c0 2024-04-26T02:21:32.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f8549ffb640 1 -- 172.21.15.87:0/3890468491 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 19356023 0 0) 0x7f854c071110 con 0x7f854c138460 2024-04-26T02:21:32.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f8549ffb640 1 -- 172.21.15.87:0/3890468491 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f854c072170 con 0x7f854c138460 2024-04-26T02:21:32.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc7b559a640 1 -- 172.21.15.87:0/1230501657 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7b002b2f0 con 0x7fc7b0037160 2024-04-26T02:21:32.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f32a747f640 1 Processor -- start 2024-04-26T02:21:32.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f32a747f640 1 -- start start 2024-04-26T02:21:32.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f9db1946640 1 -- 172.21.15.87:0/3739085866 wait complete. 2024-04-26T02:21:32.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5cd81e640 1 -- 172.21.15.87:0/756959603 >> v1:172.21.15.87:6790/0 conn(0x7fc5c80703d0 legacy=0x7fc5c8072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330159a640 1 -- 172.21.15.87:0/596276965 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3471687995 0 0) 0x7f32fc02ce80 con 0x7f32fc035240 2024-04-26T02:21:32.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330159a640 1 -- 172.21.15.87:0/596276965 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32ec003530 con 0x7f32fc035240 2024-04-26T02:21:32.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330159a640 1 -- 172.21.15.87:0/596276965 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2388817280 0 0) 0x7f32fc02d1b0 con 0x7f32fc036de0 2024-04-26T02:21:32.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330159a640 1 -- 172.21.15.87:0/596276965 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32fc02ce80 con 0x7f32fc036de0 2024-04-26T02:21:32.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5cd81e640 1 -- 172.21.15.87:0/756959603 shutdown_connections 2024-04-26T02:21:32.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5cd81e640 1 -- 172.21.15.87:0/756959603 >> 172.21.15.87:0/756959603 conn(0x7fc5c80b0210 msgr2=0x7fc5c80b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f9db1946640 1 Processor -- start 2024-04-26T02:21:32.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc7b559a640 1 -- 172.21.15.87:0/1230501657 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2483198889 0 0) 0x7fc7a0003530 con 0x7fc7b002ad40 2024-04-26T02:21:32.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc7b559a640 1 -- 172.21.15.87:0/1230501657 >> v1:172.21.15.87:6790/0 conn(0x7fc7b002c210 legacy=0x7fc7b00c7a10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T02:21:32.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc7b559a640 1 -- 172.21.15.87:0/1230501657 >> v1:172.21.15.87:6789/0 conn(0x7fc7b0037160 legacy=0x7fc7b00c1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc7b559a640 1 -- 172.21.15.87:0/1230501657 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b00c1f20 con 0x7fc7b002ad40 2024-04-26T02:21:32.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc7b559a640 1 -- 172.21.15.87:0/1230501657 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc7ac0015a0 con 0x7fc7b002ad40 2024-04-26T02:21:32.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5cd81e640 1 -- 172.21.15.87:0/756959603 shutdown_connections 2024-04-26T02:21:32.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330259c640 1 -- 172.21.15.87:0/596276965 >> v1:172.21.15.87:6789/0 conn(0x7f32fc036a00 legacy=0x7f32fc0c28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-04-26T02:21:32.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330259c640 1 -- 172.21.15.87:0/596276965 >> v1:172.21.15.87:6789/0 conn(0x7f32fc036a00 legacy=0x7f32fc0c28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T02:21:32.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330259c640 1 --1- 172.21.15.87:0/596276965 >> v1:172.21.15.87:6789/0 conn(0x7f32fc036a00 0x7f32fc0c28a0 :-1 s=OPENED pgs=91 cs=1 l=1).handle_message read tag failed 2024-04-26T02:21:32.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330259c640 1 --1- 172.21.15.87:0/596276965 >> v1:172.21.15.87:6789/0 conn(0x7f32fc036a00 0x7f32fc0c28a0 :-1 s=OPENED pgs=91 cs=1 l=1).fault on lossy channel, failing 2024-04-26T02:21:32.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f8549ffb640 1 -- 172.21.15.87:0/3890468491 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1546082191 0 0) 0x7f854c072170 con 0x7f854c138460 2024-04-26T02:21:32.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f8549ffb640 1 -- 172.21.15.87:0/3890468491 >> v1:172.21.15.122:6789/0 conn(0x7f854c072bd0 legacy=0x7f854c13e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f8549ffb640 1 -- 172.21.15.87:0/3890468491 >> v1:172.21.15.87:6789/0 conn(0x7f854c13e9c0 legacy=0x7f854c137d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f8549ffb640 1 -- 172.21.15.87:0/3890468491 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854c070510 con 0x7f854c138460 2024-04-26T02:21:32.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f8549ffb640 1 -- 172.21.15.87:0/3890468491 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f85440018c0 con 0x7f854c138460 2024-04-26T02:21:32.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f32a747f640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32a0072280 con 0x7f32a0136810 2024-04-26T02:21:32.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f32a747f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32a007ad40 con 0x7f32a0072a80 2024-04-26T02:21:32.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f32a747f640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32a007b020 con 0x7f32a006b1b0 2024-04-26T02:21:32.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc7b559a640 1 -- 172.21.15.87:0/1230501657 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc7ac0018c0 con 0x7fc7b002ad40 2024-04-26T02:21:32.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc7b559a640 1 -- 172.21.15.87:0/1230501657 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc7ac001c90 con 0x7fc7b002ad40 2024-04-26T02:21:32.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fc5cd81e640 1 -- 172.21.15.87:0/756959603 wait complete. 2024-04-26T02:21:32.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330159a640 1 -- 172.21.15.87:0/596276965 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 972516331 0 0) 0x7f32fc02ce80 con 0x7f32fc036de0 2024-04-26T02:21:32.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330159a640 1 -- 172.21.15.87:0/596276965 >> v1:172.21.15.87:6790/0 conn(0x7f32fc035240 legacy=0x7f32fc0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330159a640 1 -- 172.21.15.87:0/596276965 >> v1:172.21.15.87:6789/0 conn(0x7f32fc036a00 legacy=0x7f32fc0c28a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T02:21:32.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330159a640 1 -- 172.21.15.87:0/596276965 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32fc0c2fb0 con 0x7f32fc036de0 2024-04-26T02:21:32.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330159a640 1 -- 172.21.15.87:0/596276965 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3304050460 con 0x7f32fc036de0 2024-04-26T02:21:32.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330159a640 1 -- 172.21.15.87:0/596276965 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f330404fa40 con 0x7f32fc036de0 2024-04-26T02:21:32.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f330159a640 1 -- 172.21.15.87:0/596276965 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f330404f4d0 con 0x7f32fc036de0 2024-04-26T02:21:32.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f9db1946640 1 -- start start 2024-04-26T02:21:32.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f9db1946640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dac146340 con 0x7f9dac0a8e00 2024-04-26T02:21:32.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f9db1946640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dac0aa240 con 0x7f9dac072370 2024-04-26T02:21:32.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f9db1946640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dac1e1a20 con 0x7f9dac0af2b0 2024-04-26T02:21:32.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f32a4a1a640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f32a0072a80 0x7f32a0136100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.87:35446/0 (socket says 172.21.15.87:35446) 2024-04-26T02:21:32.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f32a5a1c640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f32a0136810 0x7f32a0132970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53704/0 (socket says 172.21.15.87:53704) 2024-04-26T02:21:32.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f32a4a1a640 1 -- 172.21.15.87:0/1988951052 learned_addr learned my addr 172.21.15.87:0/1988951052 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4043934914 0 0) 0x7f32a0072280 con 0x7f32a0136810 2024-04-26T02:21:32.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3284003530 con 0x7f32a0136810 2024-04-26T02:21:32.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3537270851 0 0) 0x7f32a007ad40 con 0x7f32a0072a80 2024-04-26T02:21:32.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32a0072280 con 0x7f32a0072a80 2024-04-26T02:21:32.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fe6890a4640 1 Processor -- start 2024-04-26T02:21:32.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7fe6890a4640 1 -- start start 2024-04-26T02:21:32.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f5ef3fff640 1 -- 172.21.15.87:0/520949763 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2508587381 0 0) 0x7f5ee8003530 con 0x7f5ef4036a60 2024-04-26T02:21:32.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f5ef3fff640 1 -- 172.21.15.87:0/520949763 >> v1:172.21.15.87:6790/0 conn(0x7f5ef40352a0 legacy=0x7f5ef40c5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f5ef3fff640 1 -- 172.21.15.87:0/520949763 >> v1:172.21.15.122:6789/0 conn(0x7f5ef40c3080 legacy=0x7f5ef40c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f5ef3fff640 1 -- 172.21.15.87:0/520949763 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef40086a0 con 0x7f5ef4036a60 2024-04-26T02:21:32.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f5ef3fff640 1 -- 172.21.15.87:0/520949763 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f5efc04fa40 con 0x7f5ef4036a60 2024-04-26T02:21:32.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5cd81e640 1 Processor -- start 2024-04-26T02:21:32.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f9db1145640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f9dac0a8e00 0x7f9dac1ef800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53714/0 (socket says 172.21.15.87:53714) 2024-04-26T02:21:32.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.609+0000 7f9db1145640 1 -- 172.21.15.87:0/3111665706 learned_addr learned my addr 172.21.15.87:0/3111665706 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f8549ffb640 1 -- 172.21.15.87:0/3890468491 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8544001b20 con 0x7f854c138460 2024-04-26T02:21:32.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f8549ffb640 1 -- 172.21.15.87:0/3890468491 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8544001ef0 con 0x7f854c138460 2024-04-26T02:21:32.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc7b759e640 1 -- 172.21.15.87:0/1230501657 >> v1:172.21.15.122:6789/0 conn(0x7fc7b002ad40 legacy=0x7fc7b00cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3423439643 0 0) 0x7f3284003530 con 0x7f32a0136810 2024-04-26T02:21:32.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f32a007ad40 con 0x7f32a0136810 2024-04-26T02:21:32.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3298001ba0 con 0x7f32a0136810 2024-04-26T02:21:32.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f5ef3fff640 1 -- 172.21.15.87:0/520949763 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5efc050460 con 0x7f5ef4036a60 2024-04-26T02:21:32.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc7b759e640 1 -- 172.21.15.87:0/1230501657 shutdown_connections 2024-04-26T02:21:32.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc7b759e640 1 -- 172.21.15.87:0/1230501657 >> 172.21.15.87:0/1230501657 conn(0x7fc7b003a530 msgr2=0x7fc7b00367f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f5ef3fff640 1 -- 172.21.15.87:0/520949763 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f5efc04f4d0 con 0x7f5ef4036a60 2024-04-26T02:21:32.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5cd81e640 1 -- start start 2024-04-26T02:21:32.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fe6890a4640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6840a8b00 con 0x7fe68406b410 2024-04-26T02:21:32.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fe6890a4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6840afc60 con 0x7fe6840b07a0 2024-04-26T02:21:32.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fe6890a4640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6840b0350 con 0x7fe6840af2b0 2024-04-26T02:21:32.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f330359e640 1 -- 172.21.15.87:0/596276965 >> v1:172.21.15.122:6789/0 conn(0x7f32fc036de0 legacy=0x7f32fc0c5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc7b759e640 1 -- 172.21.15.87:0/1230501657 shutdown_connections 2024-04-26T02:21:32.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 817898287 0 0) 0x7f32a0072280 con 0x7f32a0072a80 2024-04-26T02:21:32.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3284003530 con 0x7f32a0072a80 2024-04-26T02:21:32.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f329c001600 con 0x7f32a0072a80 2024-04-26T02:21:32.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 463869323 0 0) 0x7f9dac0aa240 con 0x7f9dac072370 2024-04-26T02:21:32.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d90003530 con 0x7f9dac072370 2024-04-26T02:21:32.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5cd81e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c8140ab0 con 0x7fc5c80af380 2024-04-26T02:21:32.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5cd81e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c80a94e0 con 0x7fc5c80ad080 2024-04-26T02:21:32.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5cd81e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c81de870 con 0x7fc5c8143c10 2024-04-26T02:21:32.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c7fff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fc5c8143c10 0x7fc5c813afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40320/0 (socket says 172.21.15.87:40320) 2024-04-26T02:21:32.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c7fff640 1 -- 172.21.15.87:0/4156267404 learned_addr learned my addr 172.21.15.87:0/4156267404 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5cc81c640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fc5c80af380 0x7fc5c813e140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53728/0 (socket says 172.21.15.87:53728) 2024-04-26T02:21:32.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 796298769 0 0) 0x7fc5c81de870 con 0x7fc5c8143c10 2024-04-26T02:21:32.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5ac003530 con 0x7fc5c8143c10 2024-04-26T02:21:32.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1286465492 0 0) 0x7fc5c80a94e0 con 0x7fc5c80ad080 2024-04-26T02:21:32.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5c81de870 con 0x7fc5c80ad080 2024-04-26T02:21:32.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2005011774 0 0) 0x7fc5c8140ab0 con 0x7fc5c80af380 2024-04-26T02:21:32.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5c80a94e0 con 0x7fc5c80af380 2024-04-26T02:21:32.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f330359e640 1 -- 172.21.15.87:0/596276965 shutdown_connections 2024-04-26T02:21:32.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f330359e640 1 -- 172.21.15.87:0/596276965 >> 172.21.15.87:0/596276965 conn(0x7f32fc005c70 msgr2=0x7f32fc006080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f330359e640 1 -- 172.21.15.87:0/596276965 shutdown_connections 2024-04-26T02:21:32.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f330359e640 1 -- 172.21.15.87:0/596276965 wait complete. 2024-04-26T02:21:32.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f330359e640 1 Processor -- start 2024-04-26T02:21:32.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2421959879 0 0) 0x7f9dac146340 con 0x7f9dac0a8e00 2024-04-26T02:21:32.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dac0aa240 con 0x7f9dac0a8e00 2024-04-26T02:21:32.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 787461540 0 0) 0x7f9dac1e1a20 con 0x7f9dac0af2b0 2024-04-26T02:21:32.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dac146340 con 0x7f9dac0af2b0 2024-04-26T02:21:32.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2528475144 0 0) 0x7f9dac0aa240 con 0x7f9dac0a8e00 2024-04-26T02:21:32.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9dac1e1a20 con 0x7f9dac0a8e00 2024-04-26T02:21:32.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9da4001c00 con 0x7f9dac0a8e00 2024-04-26T02:21:32.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 519401993 0 0) 0x7f9dac146340 con 0x7f9dac0af2b0 2024-04-26T02:21:32.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9dac0aa240 con 0x7f9dac0af2b0 2024-04-26T02:21:32.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9da0001380 con 0x7f9dac0af2b0 2024-04-26T02:21:32.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f85514c3640 1 -- 172.21.15.87:0/3890468491 >> v1:172.21.15.87:6790/0 conn(0x7f854c138460 legacy=0x7f854c149bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f85514c3640 1 -- 172.21.15.87:0/3890468491 shutdown_connections 2024-04-26T02:21:32.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f85514c3640 1 -- 172.21.15.87:0/3890468491 >> 172.21.15.87:0/3890468491 conn(0x7f854c06b1b0 msgr2=0x7f854c06b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f85514c3640 1 -- 172.21.15.87:0/3890468491 shutdown_connections 2024-04-26T02:21:32.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f85514c3640 1 -- 172.21.15.87:0/3890468491 wait complete. 2024-04-26T02:21:32.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc7b759e640 1 -- 172.21.15.87:0/1230501657 wait complete. 2024-04-26T02:21:32.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc7b759e640 1 Processor -- start 2024-04-26T02:21:32.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc7b759e640 1 -- start start 2024-04-26T02:21:32.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 <== mon.1 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3995542281 0 0) 0x7f3284003530 con 0x7f32a0072a80 2024-04-26T02:21:32.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 >> v1:172.21.15.87:6790/0 conn(0x7f32a006b1b0 legacy=0x7f32a0071880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 >> v1:172.21.15.87:6789/0 conn(0x7f32a0136810 legacy=0x7f32a0132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a0071f90 con 0x7f32a0072a80 2024-04-26T02:21:32.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f32a747f640 1 -- 172.21.15.87:0/1988951052 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f32a0136c80 con 0x7f32a0072a80 2024-04-26T02:21:32.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f32a747f640 1 -- 172.21.15.87:0/1988951052 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f32a0133080 con 0x7f32a0072a80 2024-04-26T02:21:32.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f32a747f640 1 -- 172.21.15.87:0/1988951052 --> v1:172.21.15.122:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f32a004fd20 con 0x7f32a0072a80 2024-04-26T02:21:32.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fe6888a3640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe6840b07a0 0x7fe68414b570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.87:35466/0 (socket says 172.21.15.87:35466) 2024-04-26T02:21:32.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fe6888a3640 1 -- 172.21.15.87:0/1812080223 learned_addr learned my addr 172.21.15.87:0/1812080223 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fe682ffd640 1 -- 172.21.15.87:0/1812080223 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2554233616 0 0) 0x7fe6840afc60 con 0x7fe6840b07a0 2024-04-26T02:21:32.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fe682ffd640 1 -- 172.21.15.87:0/1812080223 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe66c003530 con 0x7fe6840b07a0 2024-04-26T02:21:32.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f5f03554640 1 -- 172.21.15.87:0/520949763 >> v1:172.21.15.87:6789/0 conn(0x7f5ef4036a60 legacy=0x7f5ef40c2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f5f03554640 1 -- 172.21.15.87:0/520949763 shutdown_connections 2024-04-26T02:21:32.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f5f03554640 1 -- 172.21.15.87:0/520949763 >> 172.21.15.87:0/520949763 conn(0x7f5ef4005900 msgr2=0x7f5ef4005d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f5f03554640 1 -- 172.21.15.87:0/520949763 shutdown_connections 2024-04-26T02:21:32.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f5f03554640 1 -- 172.21.15.87:0/520949763 wait complete. 2024-04-26T02:21:32.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 382594024 0 0) 0x7f9d90003530 con 0x7f9dac072370 2024-04-26T02:21:32.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9dac146340 con 0x7f9dac072370 2024-04-26T02:21:32.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9d98001760 con 0x7f9dac072370 2024-04-26T02:21:32.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f85514c3640 1 Processor -- start 2024-04-26T02:21:32.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 <== mon.1 v1:172.21.15.122:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f329c001960 con 0x7f32a0072a80 2024-04-26T02:21:32.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 <== mon.1 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f329c003bc0 con 0x7f32a0072a80 2024-04-26T02:21:32.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f85514c3640 1 -- start start 2024-04-26T02:21:32.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 77890037 0 0) 0x7fc5c81de870 con 0x7fc5c80ad080 2024-04-26T02:21:32.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5c8140ab0 con 0x7fc5c80ad080 2024-04-26T02:21:32.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc5c0001b10 con 0x7fc5c80ad080 2024-04-26T02:21:32.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3001517231 0 0) 0x7fc5c80a94e0 con 0x7fc5c80af380 2024-04-26T02:21:32.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5c81de870 con 0x7fc5c80af380 2024-04-26T02:21:32.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc5b40017b0 con 0x7fc5c80af380 2024-04-26T02:21:32.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 <== mon.1 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3889360511 0 0) 0x7fc5c8140ab0 con 0x7fc5c80ad080 2024-04-26T02:21:32.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 >> v1:172.21.15.87:6790/0 conn(0x7fc5c8143c10 legacy=0x7fc5c813afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 >> v1:172.21.15.87:6789/0 conn(0x7fc5c80af380 legacy=0x7fc5c813e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c81dea40 con 0x7fc5c80ad080 2024-04-26T02:21:32.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f330359e640 1 -- start start 2024-04-26T02:21:32.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f330359e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32fc039f20 con 0x7f32fc036a00 2024-04-26T02:21:32.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f330359e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32fc03a620 con 0x7f32fc15f190 2024-04-26T02:21:32.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f330359e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32fc15c530 con 0x7f32fc036de0 2024-04-26T02:21:32.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2537182511 0 0) 0x7f9dac0aa240 con 0x7f9dac0af2b0 2024-04-26T02:21:32.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 >> v1:172.21.15.122:6789/0 conn(0x7f9dac072370 legacy=0x7f9dac0aba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 >> v1:172.21.15.87:6789/0 conn(0x7f9dac0a8e00 legacy=0x7f9dac1ef800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dac145fe0 con 0x7f9dac0af2b0 2024-04-26T02:21:32.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f85514c3640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f854c13b7f0 con 0x7f854c072bd0 2024-04-26T02:21:32.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f85514c3640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f854c13aa20 con 0x7f854c138460 2024-04-26T02:21:32.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f85514c3640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f854c0ae460 con 0x7f854c13e9c0 2024-04-26T02:21:32.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f854affd640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f854c072bd0 0x7f854c139820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53752/0 (socket says 172.21.15.87:53752) 2024-04-26T02:21:32.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f854affd640 1 -- 172.21.15.87:0/2786574555 learned_addr learned my addr 172.21.15.87:0/2786574555 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc7b759e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7b002f040 con 0x7fc7b0037160 2024-04-26T02:21:32.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc7b759e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7b002f2b0 con 0x7fc7b002ad40 2024-04-26T02:21:32.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc7b759e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7b00c3140 con 0x7fc7b002c210 2024-04-26T02:21:32.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc7b6d9d640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fc7b0037160 0x7fc7b016ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53746/0 (socket says 172.21.15.87:53746) 2024-04-26T02:21:32.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc7b6d9d640 1 -- 172.21.15.87:0/951236566 learned_addr learned my addr 172.21.15.87:0/951236566 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1912600289 0 0) 0x7fc7b002f040 con 0x7fc7b0037160 2024-04-26T02:21:32.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc798003530 con 0x7fc7b0037160 2024-04-26T02:21:32.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2950415271 0 0) 0x7fc7b002f2b0 con 0x7fc7b002ad40 2024-04-26T02:21:32.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7b002f040 con 0x7fc7b002ad40 2024-04-26T02:21:32.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fe682ffd640 1 -- 172.21.15.87:0/1812080223 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3986859911 0 0) 0x7fe66c003530 con 0x7fe6840b07a0 2024-04-26T02:21:32.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7fe682ffd640 1 -- 172.21.15.87:0/1812080223 >> v1:172.21.15.87:6790/0 conn(0x7fe6840af2b0 legacy=0x7fe684143210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:21:32.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f5f03554640 1 Processor -- start 2024-04-26T02:21:32.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f330259c640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f32fc036a00 0x7f32fc15be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53762/0 (socket says 172.21.15.87:53762) 2024-04-26T02:21:32.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f330259c640 1 -- 172.21.15.87:0/3603106142 learned_addr learned my addr 172.21.15.87:0/3603106142 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 <== mon.1 v1:172.21.15.122:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc5c0003980 con 0x7fc5c80ad080 2024-04-26T02:21:32.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 <== mon.1 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc5c0003cd0 con 0x7fc5c80ad080 2024-04-26T02:21:32.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2518191989 0 0) 0x7f854c13b7f0 con 0x7f854c072bd0 2024-04-26T02:21:32.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8538003530 con 0x7f854c072bd0 2024-04-26T02:21:32.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3746513899 0 0) 0x7fc798003530 con 0x7fc7b0037160 2024-04-26T02:21:32.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7b002f2b0 con 0x7fc7b0037160 2024-04-26T02:21:32.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.613+0000 7f5f03554640 1 -- start start 2024-04-26T02:21:32.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f5f03554640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef40d7360 con 0x7f5ef4036a60 2024-04-26T02:21:32.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f5f03554640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef40d7640 con 0x7f5ef40c4cd0 2024-04-26T02:21:32.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f5f03554640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef40c5160 con 0x7f5ef40352a0 2024-04-26T02:21:32.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f5f00aef640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f5ef4036a60 0x7f5ef40d5ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53768/0 (socket says 172.21.15.87:53768) 2024-04-26T02:21:32.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f5f00aef640 1 -- 172.21.15.87:0/2116150288 learned_addr learned my addr 172.21.15.87:0/2116150288 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1003019717 0 0) 0x7fc7b002f040 con 0x7fc7b002ad40 2024-04-26T02:21:32.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc798003530 con 0x7fc7b002ad40 2024-04-26T02:21:32.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc7ac001560 con 0x7fc7b0037160 2024-04-26T02:21:32.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9da00015e0 con 0x7f9dac0af2b0 2024-04-26T02:21:32.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9da0004da0 con 0x7f9dac0af2b0 2024-04-26T02:21:32.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc7a80015d0 con 0x7fc7b002ad40 2024-04-26T02:21:32.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3112182111 0 0) 0x7f32fc15c530 con 0x7f32fc036de0 2024-04-26T02:21:32.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32ec003250 con 0x7f32fc036de0 2024-04-26T02:21:32.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1680714235 0 0) 0x7f32fc03a620 con 0x7f32fc15f190 2024-04-26T02:21:32.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32fc15c530 con 0x7f32fc15f190 2024-04-26T02:21:32.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2259651385 0 0) 0x7f32ec003250 con 0x7f32fc036de0 2024-04-26T02:21:32.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f32fc03a620 con 0x7f32fc036de0 2024-04-26T02:21:32.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 <== mon.1 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f329c003da0 con 0x7f32a0072a80 2024-04-26T02:21:32.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f9db1946640 1 -- 172.21.15.87:0/3111665706 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9dac0ac190 con 0x7f9dac0af2b0 2024-04-26T02:21:32.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f9db1946640 1 -- 172.21.15.87:0/3111665706 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9dac0a8850 con 0x7f9dac0af2b0 2024-04-26T02:21:32.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 <== mon.1 v1:172.21.15.122:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7f329c081040 con 0x7f32a0072a80 2024-04-26T02:21:32.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fe682ffd640 1 -- 172.21.15.87:0/1812080223 >> v1:172.21.15.87:6789/0 conn(0x7fe68406b410 legacy=0x7fe68413cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fe682ffd640 1 -- 172.21.15.87:0/1812080223 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68413d640 con 0x7fe6840b07a0 2024-04-26T02:21:32.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fe682ffd640 1 -- 172.21.15.87:0/1812080223 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fe6740019a0 con 0x7fe6840b07a0 2024-04-26T02:21:32.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2372639429 0 0) 0x7f854c13aa20 con 0x7f854c138460 2024-04-26T02:21:32.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f854c13b7f0 con 0x7f854c138460 2024-04-26T02:21:32.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f32f40018e0 con 0x7f32fc036de0 2024-04-26T02:21:32.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3889834309 0 0) 0x7f32fc03a620 con 0x7f32fc036de0 2024-04-26T02:21:32.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 >> v1:172.21.15.122:6789/0 conn(0x7f32fc15f190 legacy=0x7f32fc16d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 >> v1:172.21.15.87:6789/0 conn(0x7f32fc036a00 legacy=0x7f32fc15be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32fc15c700 con 0x7f32fc036de0 2024-04-26T02:21:32.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f330359e640 1 -- 172.21.15.87:0/3603106142 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f32fc038c10 con 0x7f32fc036de0 2024-04-26T02:21:32.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f330359e640 1 -- 172.21.15.87:0/3603106142 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f32fc175d10 con 0x7f32fc036de0 2024-04-26T02:21:32.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 --> v1:172.21.15.122:6812/498384109 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f32840730b0 con 0x7f328406f4e0 2024-04-26T02:21:32.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 <== mon.1 v1:172.21.15.122:6789/0 9 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (unknown 77724187 0 0) 0x7f329c081420 con 0x7f32a0072a80 2024-04-26T02:21:32.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f32f40047f0 con 0x7f32fc036de0 2024-04-26T02:21:32.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f32f4004b40 con 0x7f32fc036de0 2024-04-26T02:21:32.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f9db1946640 1 -- 172.21.15.87:0/3111665706 --> v1:172.21.15.87:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f9dac062e10 con 0x7f9dac0af2b0 2024-04-26T02:21:32.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fe682ffd640 1 -- 172.21.15.87:0/1812080223 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe674001bb0 con 0x7fe6840b07a0 2024-04-26T02:21:32.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fe682ffd640 1 -- 172.21.15.87:0/1812080223 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fe674001fc0 con 0x7fe6840b07a0 2024-04-26T02:21:32.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3305268528 0 0) 0x7f8538003530 con 0x7f854c072bd0 2024-04-26T02:21:32.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc5cd81e640 1 -- 172.21.15.87:0/4156267404 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c813b6d0 con 0x7fc5c80ad080 2024-04-26T02:21:32.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc5cd81e640 1 -- 172.21.15.87:0/4156267404 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc5c81df0c0 con 0x7fc5c80ad080 2024-04-26T02:21:32.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc5cd81e640 1 -- 172.21.15.87:0/4156267404 --> v1:172.21.15.122:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fc5c8140ee0 con 0x7fc5c80ad080 2024-04-26T02:21:32.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f854c13aa20 con 0x7f854c072bd0 2024-04-26T02:21:32.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f853c001a40 con 0x7f854c072bd0 2024-04-26T02:21:32.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fe6890a4640 1 -- 172.21.15.87:0/1812080223 >> v1:172.21.15.122:6789/0 conn(0x7fe6840b07a0 legacy=0x7fe68414b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1517416841 0 0) 0x7f854c13aa20 con 0x7f854c072bd0 2024-04-26T02:21:32.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 >> v1:172.21.15.87:6790/0 conn(0x7f854c13e9c0 legacy=0x7f854c1dd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 >> v1:172.21.15.122:6789/0 conn(0x7f854c138460 legacy=0x7f854c13a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854c0ad9c0 con 0x7f854c072bd0 2024-04-26T02:21:32.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fe6890a4640 1 -- 172.21.15.87:0/1812080223 shutdown_connections 2024-04-26T02:21:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fe6890a4640 1 -- 172.21.15.87:0/1812080223 >> 172.21.15.87:0/1812080223 conn(0x7fe6840b1220 msgr2=0x7fe6840b1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f330359e640 1 -- 172.21.15.87:0/3603106142 --> v1:172.21.15.87:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f32e8000f90 con 0x7f32fc036de0 2024-04-26T02:21:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f32f4004d60 con 0x7f32fc036de0 2024-04-26T02:21:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f85514c3640 1 -- 172.21.15.87:0/2786574555 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f854c1d2cf0 con 0x7f854c072bd0 2024-04-26T02:21:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f85514c3640 1 -- 172.21.15.87:0/2786574555 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f854c1d31b0 con 0x7f854c072bd0 2024-04-26T02:21:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f85514c3640 1 -- 172.21.15.87:0/2786574555 --> v1:172.21.15.87:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f854c139400 con 0x7f854c072bd0 2024-04-26T02:21:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 <== mon.1 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2406598511 0 0) 0x7fc798003530 con 0x7fc7b002ad40 2024-04-26T02:21:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 >> v1:172.21.15.87:6790/0 conn(0x7fc7b002c210 legacy=0x7fc7b01614e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 >> v1:172.21.15.87:6789/0 conn(0x7fc7b0037160 legacy=0x7fc7b016ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b00c3310 con 0x7fc7b002ad40 2024-04-26T02:21:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7b759e640 1 -- 172.21.15.87:0/951236566 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b0161bf0 con 0x7fc7b002ad40 2024-04-26T02:21:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7b759e640 1 -- 172.21.15.87:0/951236566 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc7b002de80 con 0x7fc7b002ad40 2024-04-26T02:21:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 <== mon.1 v1:172.21.15.122:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc7a8001c20 con 0x7fc7b002ad40 2024-04-26T02:21:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 <== mon.1 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc7a80047f0 con 0x7fc7b002ad40 2024-04-26T02:21:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 520904226 0 0) 0x7f5ef40d7360 con 0x7f5ef4036a60 2024-04-26T02:21:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ee0003530 con 0x7f5ef4036a60 2024-04-26T02:21:32.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4205660768 0 0) 0x7f5ef40d7640 con 0x7f5ef40c4cd0 2024-04-26T02:21:32.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ef40d7360 con 0x7f5ef40c4cd0 2024-04-26T02:21:32.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2077016315 0 0) 0x7f5ef40c5160 con 0x7f5ef40352a0 2024-04-26T02:21:32.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.617+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ef40d7640 con 0x7f5ef40352a0 2024-04-26T02:21:32.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 151605251 0 0) 0x7f5ee0003530 con 0x7f5ef4036a60 2024-04-26T02:21:32.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ef40c5160 con 0x7f5ef4036a60 2024-04-26T02:21:32.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f5ef8001840 con 0x7f5ef4036a60 2024-04-26T02:21:32.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7fe6890a4640 1 -- 172.21.15.87:0/1812080223 shutdown_connections 2024-04-26T02:21:32.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7f32f4080fe0 con 0x7f32fc036de0 2024-04-26T02:21:32.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 <== mon.1 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7fc5c00193f0 con 0x7fc5c80ad080 2024-04-26T02:21:32.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 <== mon.1 v1:172.21.15.122:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7fc5c0080010 con 0x7fc5c80ad080 2024-04-26T02:21:32.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 --> v1:172.21.15.87:6804/815800445 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5ac073140 con 0x7fc5ac06f570 2024-04-26T02:21:32.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 <== mon.1 v1:172.21.15.122:6789/0 9 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (unknown 77724187 0 0) 0x7fc5c00803e0 con 0x7fc5c80ad080 2024-04-26T02:21:32.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 --> v1:172.21.15.87:6800/174846219 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f32ec073440 con 0x7f32ec06f870 2024-04-26T02:21:32.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 <== osd.5 v1:172.21.15.122:6812/498384109 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f32840730b0 con 0x7f328406f4e0 2024-04-26T02:21:32.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1917446812 0 0) 0x7f5ef40c5160 con 0x7f5ef4036a60 2024-04-26T02:21:32.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 >> v1:172.21.15.87:6790/0 conn(0x7f5ef40352a0 legacy=0x7f5ef40d58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 >> v1:172.21.15.122:6789/0 conn(0x7f5ef40c4cd0 legacy=0x7f5ef416b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef40d8010 con 0x7f5ef4036a60 2024-04-26T02:21:32.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f5f03554640 1 -- 172.21.15.87:0/2116150288 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5ef40d5240 con 0x7f5ef4036a60 2024-04-26T02:21:32.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f5f03554640 1 -- 172.21.15.87:0/2116150288 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5ef416cc50 con 0x7f5ef4036a60 2024-04-26T02:21:32.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f5f03554640 1 -- 172.21.15.87:0/2116150288 --> v1:172.21.15.87:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f5ef4004ce0 con 0x7f5ef4036a60 2024-04-26T02:21:32.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ef8003600 con 0x7f5ef4036a60 2024-04-26T02:21:32.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f5ef8003950 con 0x7f5ef4036a60 2024-04-26T02:21:32.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7fe6890a4640 1 -- 172.21.15.87:0/1812080223 wait complete. 2024-04-26T02:21:32.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7fe6890a4640 1 Processor -- start 2024-04-26T02:21:32.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f854c139400 con 0x7f854c072bd0 2024-04-26T02:21:32.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 <== osd.1 v1:172.21.15.87:6800/174846219 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f32ec073440 con 0x7f32ec06f870 2024-04-26T02:21:32.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 <== osd.0 v1:172.21.15.87:6804/815800445 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc5ac073140 con 0x7fc5ac06f570 2024-04-26T02:21:32.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 <== mon.1 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7fc7a8004990 con 0x7fc7b002ad40 2024-04-26T02:21:32.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (unknown 77724187 0 0) 0x7f32f404db10 con 0x7f32fc036de0 2024-04-26T02:21:32.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 <== mon.1 v1:172.21.15.122:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7fc7a8080d80 con 0x7fc7b002ad40 2024-04-26T02:21:32.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.621+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f853c004aa0 con 0x7f854c072bd0 2024-04-26T02:21:32.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fc7b759e640 1 -- 172.21.15.87:0/951236566 --> v1:172.21.15.122:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fc79c000f90 con 0x7fc7b002ad40 2024-04-26T02:21:32.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f9da001a320 con 0x7f9dac0af2b0 2024-04-26T02:21:32.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 --> v1:172.21.15.122:6800/850918874 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc798073010 con 0x7fc79806f3a0 2024-04-26T02:21:32.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f5ef8003af0 con 0x7f5ef4036a60 2024-04-26T02:21:32.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe6890a4640 1 -- start start 2024-04-26T02:21:32.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f853c01a1c0 con 0x7f854c072bd0 2024-04-26T02:21:32.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7f853c081000 con 0x7f854c072bd0 2024-04-26T02:21:32.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 --> v1:172.21.15.87:6812/3926229128 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8538073150 con 0x7f853806f580 2024-04-26T02:21:32.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (unknown 77724187 0 0) 0x7f853c0813d0 con 0x7f854c072bd0 2024-04-26T02:21:32.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe6890a4640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6841e1320 con 0x7fe6841e7610 2024-04-26T02:21:32.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe6890a4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe684141520 con 0x7fe68406b410 2024-04-26T02:21:32.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe6890a4640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe684142f60 con 0x7fe6840b07a0 2024-04-26T02:21:32.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe6837fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fe6840b07a0 0x7fe6841e0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40362/0 (socket says 172.21.15.87:40362) 2024-04-26T02:21:32.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe6837fe640 1 -- 172.21.15.87:0/4171948194 learned_addr learned my addr 172.21.15.87:0/4171948194 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7f9da0080c10 con 0x7f9dac0af2b0 2024-04-26T02:21:32.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 --> v1:172.21.15.87:6808/2992851618 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d900732e0 con 0x7f9d9006f710 2024-04-26T02:21:32.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (unknown 77724187 0 0) 0x7f9da0080ff0 con 0x7f9dac0af2b0 2024-04-26T02:21:32.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 <== mon.1 v1:172.21.15.122:6789/0 9 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (unknown 77724187 0 0) 0x7fc7a804d8b0 con 0x7fc7b002ad40 2024-04-26T02:21:32.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1743096256 0 0) 0x7fe684142f60 con 0x7fe6840b07a0 2024-04-26T02:21:32.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe670003560 con 0x7fe6840b07a0 2024-04-26T02:21:32.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3750245841 0 0) 0x7fe684141520 con 0x7fe68406b410 2024-04-26T02:21:32.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe684142f60 con 0x7fe68406b410 2024-04-26T02:21:32.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 102341608 0 0) 0x7fe670003560 con 0x7fe6840b07a0 2024-04-26T02:21:32.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe684141520 con 0x7fe6840b07a0 2024-04-26T02:21:32.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fe678001920 con 0x7fe6840b07a0 2024-04-26T02:21:32.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2387734932 0 0) 0x7fe684141520 con 0x7fe6840b07a0 2024-04-26T02:21:32.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 >> v1:172.21.15.122:6789/0 conn(0x7fe68406b410 legacy=0x7fe6841e6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 >> v1:172.21.15.87:6789/0 conn(0x7fe6841e7610 legacy=0x7fe6841e0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe684143240 con 0x7fe6840b07a0 2024-04-26T02:21:32.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7f5ef8080f50 con 0x7f5ef4036a60 2024-04-26T02:21:32.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 --> v1:172.21.15.122:6804/7687745 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ee0073260 con 0x7f5ee006f5f0 2024-04-26T02:21:32.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (unknown 77724187 0 0) 0x7f5ef8081320 con 0x7f5ef4036a60 2024-04-26T02:21:32.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe6890a4640 1 -- 172.21.15.87:0/4171948194 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe684141ce0 con 0x7fe6840b07a0 2024-04-26T02:21:32.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe6890a4640 1 -- 172.21.15.87:0/4171948194 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe6840b42d0 con 0x7fe6840b07a0 2024-04-26T02:21:32.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe678003600 con 0x7fe6840b07a0 2024-04-26T02:21:32.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fe678003950 con 0x7fe6840b07a0 2024-04-26T02:21:32.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== osd.2 v1:172.21.15.87:6808/2992851618 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9d900732e0 con 0x7f9d9006f710 2024-04-26T02:21:32.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7fe678003af0 con 0x7fe6840b07a0 2024-04-26T02:21:32.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.625+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 <== osd.7 v1:172.21.15.122:6804/7687745 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5ee0073260 con 0x7f5ee006f5f0 2024-04-26T02:21:32.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.629+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7fe6780810f0 con 0x7fe6840b07a0 2024-04-26T02:21:32.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.629+0000 7fe6890a4640 1 -- 172.21.15.87:0/4171948194 --> v1:172.21.15.122:6805/662791991 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6640053b0 con 0x7fe664001640 2024-04-26T02:21:32.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.629+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 <== osd.3 v1:172.21.15.87:6812/3926229128 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8538073150 con 0x7f853806f580 2024-04-26T02:21:32.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.633+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 <== osd.4 v1:172.21.15.122:6800/850918874 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc798073010 con 0x7fc79806f3a0 2024-04-26T02:21:32.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.633+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 <== osd.6 v1:172.21.15.122:6805/662791991 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe6640053b0 con 0x7fe664001640 2024-04-26T02:21:32.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.649+0000 7f330359e640 1 -- 172.21.15.87:0/3603106142 --> v1:172.21.15.87:6800/174846219 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f32e8002c80 con 0x7f32ec06f870 2024-04-26T02:21:32.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.649+0000 7f32a747f640 1 -- 172.21.15.87:0/1988951052 --> v1:172.21.15.122:6812/498384109 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f32a006b590 con 0x7f328406f4e0 2024-04-26T02:21:32.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.649+0000 7f32f2ffd640 1 -- 172.21.15.87:0/3603106142 <== osd.1 v1:172.21.15.87:6800/174846219 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 1046766345) 0x7f32e8002c80 con 0x7f32ec06f870 2024-04-26T02:21:32.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.649+0000 7f3291ffb640 1 -- 172.21.15.87:0/1988951052 <== osd.5 v1:172.21.15.122:6812/498384109 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3423592458) 0x7f32a006b590 con 0x7f328406f4e0 2024-04-26T02:21:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.649+0000 7f328b7fe640 1 -- 172.21.15.87:0/1988951052 >> v1:172.21.15.122:6812/498384109 conn(0x7f328406f4e0 legacy=0x7f3284071900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f328b7fe640 1 -- 172.21.15.87:0/1988951052 >> v1:172.21.15.122:6816/416193625 conn(0x7f32840683c0 legacy=0x7f328406a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f328b7fe640 1 -- 172.21.15.87:0/1988951052 >> v1:172.21.15.122:6789/0 conn(0x7f32a0072a80 legacy=0x7f32a0136100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f32a5a1c640 1 -- 172.21.15.87:0/1988951052 reap_dead start 2024-04-26T02:21:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f328b7fe640 1 -- 172.21.15.87:0/1988951052 shutdown_connections 2024-04-26T02:21:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f328b7fe640 1 -- 172.21.15.87:0/1988951052 >> 172.21.15.87:0/1988951052 conn(0x7f32a00b4a10 msgr2=0x7f32a00b4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f328b7fe640 1 -- 172.21.15.87:0/1988951052 shutdown_connections 2024-04-26T02:21:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f9db1946640 1 -- 172.21.15.87:0/3111665706 --> v1:172.21.15.87:6808/2992851618 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9dac0ac6c0 con 0x7f9d9006f710 2024-04-26T02:21:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f328b7fe640 1 -- 172.21.15.87:0/1988951052 wait complete. 2024-04-26T02:21:32.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f85514c3640 1 -- 172.21.15.87:0/2786574555 --> v1:172.21.15.87:6812/3926229128 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f854c062e10 con 0x7f853806f580 2024-04-26T02:21:32.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f32f0ff9640 1 -- 172.21.15.87:0/3603106142 >> v1:172.21.15.87:6800/174846219 conn(0x7f32ec06f870 legacy=0x7f32ec071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f9da97fa640 1 -- 172.21.15.87:0/3111665706 <== osd.2 v1:172.21.15.87:6808/2992851618 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 700469118) 0x7f9dac0ac6c0 con 0x7f9d9006f710 2024-04-26T02:21:32.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f85337fe640 1 -- 172.21.15.87:0/2786574555 <== osd.3 v1:172.21.15.87:6812/3926229128 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 2451364910) 0x7f854c062e10 con 0x7f853806f580 2024-04-26T02:21:32.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f32f0ff9640 1 -- 172.21.15.87:0/3603106142 >> v1:172.21.15.122:6816/416193625 conn(0x7f32ec068750 legacy=0x7f32ec06ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f32f0ff9640 1 -- 172.21.15.87:0/3603106142 >> v1:172.21.15.87:6790/0 conn(0x7f32fc036de0 legacy=0x7f32fc16cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f3302d9d640 1 -- 172.21.15.87:0/3603106142 reap_dead start 2024-04-26T02:21:32.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f9db1946640 1 -- 172.21.15.87:0/3111665706 >> v1:172.21.15.87:6808/2992851618 conn(0x7f9d9006f710 legacy=0x7f9d90071b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f32f0ff9640 1 -- 172.21.15.87:0/3603106142 shutdown_connections 2024-04-26T02:21:32.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f32f0ff9640 1 -- 172.21.15.87:0/3603106142 >> 172.21.15.87:0/3603106142 conn(0x7f32fc005c70 msgr2=0x7f32fc008c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f85514c3640 1 -- 172.21.15.87:0/2786574555 >> v1:172.21.15.87:6812/3926229128 conn(0x7f853806f580 legacy=0x7f85380719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f32f0ff9640 1 -- 172.21.15.87:0/3603106142 shutdown_connections 2024-04-26T02:21:32.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f32f0ff9640 1 -- 172.21.15.87:0/3603106142 wait complete. 2024-04-26T02:21:32.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f9db1946640 1 -- 172.21.15.87:0/3111665706 >> v1:172.21.15.122:6816/416193625 conn(0x7f9d90068680 legacy=0x7f9d9006ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f85514c3640 1 -- 172.21.15.87:0/2786574555 >> v1:172.21.15.122:6816/416193625 conn(0x7f8538068460 legacy=0x7f853806a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f9db1946640 1 -- 172.21.15.87:0/3111665706 >> v1:172.21.15.87:6790/0 conn(0x7f9dac0af2b0 legacy=0x7f9dac1e4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f9db1145640 1 -- 172.21.15.87:0/3111665706 reap_dead start 2024-04-26T02:21:32.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f85514c3640 1 -- 172.21.15.87:0/2786574555 >> v1:172.21.15.87:6789/0 conn(0x7f854c072bd0 legacy=0x7f854c139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f854b7fe640 1 -- 172.21.15.87:0/2786574555 reap_dead start 2024-04-26T02:21:32.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f85514c3640 1 -- 172.21.15.87:0/2786574555 shutdown_connections 2024-04-26T02:21:32.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f85514c3640 1 -- 172.21.15.87:0/2786574555 >> 172.21.15.87:0/2786574555 conn(0x7f854c06b1b0 msgr2=0x7f854c0b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f85514c3640 1 -- 172.21.15.87:0/2786574555 shutdown_connections 2024-04-26T02:21:32.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f85514c3640 1 -- 172.21.15.87:0/2786574555 wait complete. 2024-04-26T02:21:32.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7fc5cd81e640 1 -- 172.21.15.87:0/4156267404 --> v1:172.21.15.87:6804/815800445 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc5c80a96b0 con 0x7fc5ac06f570 2024-04-26T02:21:32.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc5c57fa640 1 -- 172.21.15.87:0/4156267404 <== osd.0 v1:172.21.15.87:6804/815800445 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3423592458) 0x7fc5c80a96b0 con 0x7fc5ac06f570 2024-04-26T02:21:32.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc5cd81e640 1 -- 172.21.15.87:0/4156267404 >> v1:172.21.15.87:6804/815800445 conn(0x7fc5ac06f570 legacy=0x7fc5ac071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc5cd81e640 1 -- 172.21.15.87:0/4156267404 >> v1:172.21.15.122:6816/416193625 conn(0x7fc5ac0684e0 legacy=0x7fc5ac06a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc5cd81e640 1 -- 172.21.15.87:0/4156267404 >> v1:172.21.15.122:6789/0 conn(0x7fc5c80ad080 legacy=0x7fc5c81e6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc5cd01d640 1 -- 172.21.15.87:0/4156267404 reap_dead start 2024-04-26T02:21:32.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc5cd81e640 1 -- 172.21.15.87:0/4156267404 shutdown_connections 2024-04-26T02:21:32.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc5cd81e640 1 -- 172.21.15.87:0/4156267404 >> 172.21.15.87:0/4156267404 conn(0x7fc5c80b0210 msgr2=0x7fc5c80b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc5cd81e640 1 -- 172.21.15.87:0/4156267404 shutdown_connections 2024-04-26T02:21:32.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc5cd81e640 1 -- 172.21.15.87:0/4156267404 wait complete. 2024-04-26T02:21:32.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f9db1946640 1 -- 172.21.15.87:0/3111665706 shutdown_connections 2024-04-26T02:21:32.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.653+0000 7f9db1946640 1 -- 172.21.15.87:0/3111665706 >> 172.21.15.87:0/3111665706 conn(0x7f9dac06b8c0 msgr2=0x7f9dac13f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7f9db1946640 1 -- 172.21.15.87:0/3111665706 shutdown_connections 2024-04-26T02:21:32.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7f9db1946640 1 -- 172.21.15.87:0/3111665706 wait complete. 2024-04-26T02:21:32.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc7b759e640 1 -- 172.21.15.87:0/951236566 --> v1:172.21.15.122:6800/850918874 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc79c002c60 con 0x7fc79806f3a0 2024-04-26T02:21:32.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc7a6ffd640 1 -- 172.21.15.87:0/951236566 <== osd.4 v1:172.21.15.122:6800/850918874 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3423592458) 0x7fc79c002c60 con 0x7fc79806f3a0 2024-04-26T02:21:32.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc7b759e640 1 -- 172.21.15.87:0/951236566 >> v1:172.21.15.122:6800/850918874 conn(0x7fc79806f3a0 legacy=0x7fc7980717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7f5f03554640 1 -- 172.21.15.87:0/2116150288 --> v1:172.21.15.122:6804/7687745 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5ef4008b40 con 0x7f5ee006f5f0 2024-04-26T02:21:32.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7f5ef1ffb640 1 -- 172.21.15.87:0/2116150288 <== osd.7 v1:172.21.15.122:6804/7687745 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 1046766345) 0x7f5ef4008b40 con 0x7f5ee006f5f0 2024-04-26T02:21:32.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7f5f03554640 1 -- 172.21.15.87:0/2116150288 >> v1:172.21.15.122:6804/7687745 conn(0x7f5ee006f5f0 legacy=0x7f5ee0071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7f5f03554640 1 -- 172.21.15.87:0/2116150288 >> v1:172.21.15.122:6816/416193625 conn(0x7f5ee0068560 legacy=0x7f5ee006aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7f5f03554640 1 -- 172.21.15.87:0/2116150288 >> v1:172.21.15.87:6789/0 conn(0x7f5ef4036a60 legacy=0x7f5ef40d5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7f5f01af1640 1 -- 172.21.15.87:0/2116150288 reap_dead start 2024-04-26T02:21:32.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7f5f03554640 1 -- 172.21.15.87:0/2116150288 shutdown_connections 2024-04-26T02:21:32.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7f5f03554640 1 -- 172.21.15.87:0/2116150288 >> 172.21.15.87:0/2116150288 conn(0x7f5ef4005900 msgr2=0x7f5ef40c2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7f5f03554640 1 -- 172.21.15.87:0/2116150288 shutdown_connections 2024-04-26T02:21:32.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7f5f03554640 1 -- 172.21.15.87:0/2116150288 wait complete. 2024-04-26T02:21:32.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc7b759e640 1 -- 172.21.15.87:0/951236566 >> v1:172.21.15.122:6816/416193625 conn(0x7fc798068310 legacy=0x7fc79806a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc7b759e640 1 -- 172.21.15.87:0/951236566 >> v1:172.21.15.122:6789/0 conn(0x7fc7b002ad40 legacy=0x7fc7b00c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc7b6d9d640 1 -- 172.21.15.87:0/951236566 reap_dead start 2024-04-26T02:21:32.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc7b759e640 1 -- 172.21.15.87:0/951236566 shutdown_connections 2024-04-26T02:21:32.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.657+0000 7fc7b759e640 1 -- 172.21.15.87:0/951236566 >> 172.21.15.87:0/951236566 conn(0x7fc7b003a530 msgr2=0x7fc7b0035c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.661+0000 7fc7b759e640 1 -- 172.21.15.87:0/951236566 shutdown_connections 2024-04-26T02:21:32.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.661+0000 7fc7b759e640 1 -- 172.21.15.87:0/951236566 wait complete. 2024-04-26T02:21:32.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.665+0000 7fe6890a4640 1 -- 172.21.15.87:0/4171948194 --> v1:172.21.15.122:6805/662791991 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe664007040 con 0x7fe664001640 2024-04-26T02:21:32.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.665+0000 7fe680ff9640 1 -- 172.21.15.87:0/4171948194 <== osd.6 v1:172.21.15.122:6805/662791991 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 700469118) 0x7fe664007040 con 0x7fe664001640 2024-04-26T02:21:32.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.669+0000 7fe6890a4640 1 -- 172.21.15.87:0/4171948194 >> v1:172.21.15.122:6805/662791991 conn(0x7fe664001640 legacy=0x7fe664003b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.669+0000 7fe6890a4640 1 -- 172.21.15.87:0/4171948194 >> v1:172.21.15.122:6816/416193625 conn(0x7fe670068490 legacy=0x7fe67006a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.669+0000 7fe6890a4640 1 -- 172.21.15.87:0/4171948194 >> v1:172.21.15.87:6790/0 conn(0x7fe6840b07a0 legacy=0x7fe6841e0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.669+0000 7fe6888a3640 1 -- 172.21.15.87:0/4171948194 reap_dead start 2024-04-26T02:21:32.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.669+0000 7fe6890a4640 1 -- 172.21.15.87:0/4171948194 shutdown_connections 2024-04-26T02:21:32.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.669+0000 7fe6890a4640 1 -- 172.21.15.87:0/4171948194 >> 172.21.15.87:0/4171948194 conn(0x7fe6840b1220 msgr2=0x7fe684139070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.669+0000 7fe6890a4640 1 -- 172.21.15.87:0/4171948194 shutdown_connections 2024-04-26T02:21:32.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.669+0000 7fe6890a4640 1 -- 172.21.15.87:0/4171948194 wait complete. 2024-04-26T02:21:32.689 INFO:teuthology.orchestra.run.smithi087.stdout:60129542146 2024-04-26T02:21:32.690 INFO:teuthology.orchestra.run.smithi087.stdout:60129542147 2024-04-26T02:21:32.691 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-26T02:21:32.691 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T02:21:32.692 INFO:teuthology.orchestra.run.smithi087.stdout:60129542147 2024-04-26T02:21:32.692 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T02:21:32.693 INFO:teuthology.orchestra.run.smithi087.stdout:60129542148 2024-04-26T02:21:32.693 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T02:21:32.694 INFO:teuthology.orchestra.run.smithi087.stdout:55834574852 2024-04-26T02:21:32.695 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T02:21:32.696 INFO:teuthology.orchestra.run.smithi087.stdout:60129542147 2024-04-26T02:21:32.697 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-26T02:21:32.698 INFO:teuthology.orchestra.run.smithi087.stdout:60129542146 2024-04-26T02:21:32.698 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-26T02:21:32.700 INFO:teuthology.orchestra.run.smithi087.stdout:55834574852 2024-04-26T02:21:32.700 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-26T02:21:32.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.837+0000 7f6aa3fff640 1 Processor -- start 2024-04-26T02:21:32.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.837+0000 7f6aa3fff640 1 -- start start 2024-04-26T02:21:32.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa3fff640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a9c02c190 con 0x7f6a9c034850 2024-04-26T02:21:32.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa3fff640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a9c02c400 con 0x7f6a9c0cc480 2024-04-26T02:21:32.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa3fff640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a9c0c9890 con 0x7f6a9c0c9450 2024-04-26T02:21:32.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa2ffd640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f6a9c034850 0x7f6a9c0c8d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53774/0 (socket says 172.21.15.87:53774) 2024-04-26T02:21:32.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa2ffd640 1 -- 172.21.15.87:0/1522927796 learned_addr learned my addr 172.21.15.87:0/1522927796 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa27fc640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f6a9c0c9450 0x7f6a9c0cbd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40364/0 (socket says 172.21.15.87:40364) 2024-04-26T02:21:32.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a65098640 1 Processor -- start 2024-04-26T02:21:32.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa1ffb640 1 -- 172.21.15.87:0/1522927796 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2097651495 0 0) 0x7f6a9c02c190 con 0x7f6a9c034850 2024-04-26T02:21:32.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa1ffb640 1 -- 172.21.15.87:0/1522927796 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a8c003530 con 0x7f6a9c034850 2024-04-26T02:21:32.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa1ffb640 1 -- 172.21.15.87:0/1522927796 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3329341472 0 0) 0x7f6aa4050460 con 0x7f6a9c0c9450 2024-04-26T02:21:32.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa1ffb640 1 -- 172.21.15.87:0/1522927796 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a9c02c190 con 0x7f6a9c0c9450 2024-04-26T02:21:32.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa1ffb640 1 -- 172.21.15.87:0/1522927796 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3678514721 0 0) 0x7f6a94001120 con 0x7f6a9c0cc480 2024-04-26T02:21:32.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa1ffb640 1 -- 172.21.15.87:0/1522927796 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6aa4050460 con 0x7f6a9c0cc480 2024-04-26T02:21:32.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa1ffb640 1 -- 172.21.15.87:0/1522927796 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 243269942 0 0) 0x7f6a8c003530 con 0x7f6a9c034850 2024-04-26T02:21:32.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa1ffb640 1 -- 172.21.15.87:0/1522927796 >> v1:172.21.15.87:6790/0 conn(0x7f6a9c0c9450 legacy=0x7f6a9c0cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa1ffb640 1 -- 172.21.15.87:0/1522927796 >> v1:172.21.15.122:6789/0 conn(0x7f6a9c0cc480 legacy=0x7f6a9c0d7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa1ffb640 1 -- 172.21.15.87:0/1522927796 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9c0cc8d0 con 0x7f6a9c034850 2024-04-26T02:21:32.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa1ffb640 1 -- 172.21.15.87:0/1522927796 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f6a98001620 con 0x7f6a9c034850 2024-04-26T02:21:32.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa1ffb640 1 -- 172.21.15.87:0/1522927796 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a98001880 con 0x7f6a9c034850 2024-04-26T02:21:32.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa1ffb640 1 -- 172.21.15.87:0/1522927796 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f6a98001c50 con 0x7f6a9c034850 2024-04-26T02:21:32.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a65098640 1 -- start start 2024-04-26T02:21:32.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a65098640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a60071880 con 0x7f9a6006b410 2024-04-26T02:21:32.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a65098640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a60070010 con 0x7f9a600b0c20 2024-04-26T02:21:32.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a65098640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a600702f0 con 0x7f9a60072340 2024-04-26T02:21:32.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a64897640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f9a60072340 0x7f9a60151ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40372/0 (socket says 172.21.15.87:40372) 2024-04-26T02:21:32.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a64897640 1 -- 172.21.15.87:0/3832458778 learned_addr learned my addr 172.21.15.87:0/3832458778 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a5effd640 1 -- 172.21.15.87:0/3832458778 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 283674319 0 0) 0x7f9a600702f0 con 0x7f9a60072340 2024-04-26T02:21:32.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a5effd640 1 -- 172.21.15.87:0/3832458778 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a48003530 con 0x7f9a60072340 2024-04-26T02:21:32.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a5effd640 1 -- 172.21.15.87:0/3832458778 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1420142889 0 0) 0x7f9a60070010 con 0x7f9a600b0c20 2024-04-26T02:21:32.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a5effd640 1 -- 172.21.15.87:0/3832458778 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a600702f0 con 0x7f9a600b0c20 2024-04-26T02:21:32.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a5effd640 1 -- 172.21.15.87:0/3832458778 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3349313893 0 0) 0x7f9a48003530 con 0x7f9a60072340 2024-04-26T02:21:32.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a5effd640 1 -- 172.21.15.87:0/3832458778 >> v1:172.21.15.122:6789/0 conn(0x7f9a600b0c20 legacy=0x7f9a60071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a5effd640 1 -- 172.21.15.87:0/3832458778 >> v1:172.21.15.87:6789/0 conn(0x7f9a6006b410 legacy=0x7f9a60143700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:21:32.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a5effd640 1 -- 172.21.15.87:0/3832458778 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a600b0420 con 0x7f9a60072340 2024-04-26T02:21:32.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a5effd640 1 -- 172.21.15.87:0/3832458778 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9a540015a0 con 0x7f9a60072340 2024-04-26T02:21:32.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa3fff640 1 -- 172.21.15.87:0/1522927796 >> v1:172.21.15.87:6789/0 conn(0x7f6a9c034850 legacy=0x7f6a9c0c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a5effd640 1 -- 172.21.15.87:0/3832458778 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9a54001800 con 0x7f9a60072340 2024-04-26T02:21:32.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f9a5effd640 1 -- 172.21.15.87:0/3832458778 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9a54001b50 con 0x7f9a60072340 2024-04-26T02:21:32.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa3fff640 1 -- 172.21.15.87:0/1522927796 shutdown_connections 2024-04-26T02:21:32.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa3fff640 1 -- 172.21.15.87:0/1522927796 >> 172.21.15.87:0/1522927796 conn(0x7f6a9c005900 msgr2=0x7f6a9c005d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa3fff640 1 -- 172.21.15.87:0/1522927796 shutdown_connections 2024-04-26T02:21:32.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f950bfff640 1 Processor -- start 2024-04-26T02:21:32.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f3f71fee640 1 Processor -- start 2024-04-26T02:21:32.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f9a65098640 1 -- 172.21.15.87:0/3832458778 >> v1:172.21.15.87:6790/0 conn(0x7f9a60072340 legacy=0x7f9a60151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f9a65098640 1 -- 172.21.15.87:0/3832458778 shutdown_connections 2024-04-26T02:21:32.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f9a65098640 1 -- 172.21.15.87:0/3832458778 >> 172.21.15.87:0/3832458778 conn(0x7f9a600b1220 msgr2=0x7f9a600b1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.841+0000 7f6aa3fff640 1 -- 172.21.15.87:0/1522927796 wait complete. 2024-04-26T02:21:32.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f6aa3fff640 1 Processor -- start 2024-04-26T02:21:32.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f6aa3fff640 1 -- start start 2024-04-26T02:21:32.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f3f71fee640 1 -- start start 2024-04-26T02:21:32.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f950bfff640 1 -- start start 2024-04-26T02:21:32.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f950bfff640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f950402b320 con 0x7f9504033090 2024-04-26T02:21:32.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f950bfff640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f950402a310 con 0x7f95040cf8e0 2024-04-26T02:21:32.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f950bfff640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f950402c190 con 0x7f9504006a80 2024-04-26T02:21:32.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f950affd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f9504006a80 0x7f95040c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40384/0 (socket says 172.21.15.87:40384) 2024-04-26T02:21:32.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f950affd640 1 -- 172.21.15.87:0/2992029654 learned_addr learned my addr 172.21.15.87:0/2992029654 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f9509ffb640 1 -- 172.21.15.87:0/2992029654 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4114741781 0 0) 0x7f950402c190 con 0x7f9504006a80 2024-04-26T02:21:32.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f9509ffb640 1 -- 172.21.15.87:0/2992029654 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94f4014540 con 0x7f9504006a80 2024-04-26T02:21:32.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f9509ffb640 1 -- 172.21.15.87:0/2992029654 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2371230757 0 0) 0x7f94f4014540 con 0x7f9504006a80 2024-04-26T02:21:32.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f9509ffb640 1 -- 172.21.15.87:0/2992029654 >> v1:172.21.15.122:6789/0 conn(0x7f95040cf8e0 legacy=0x7f95040d7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f9509ffb640 1 -- 172.21.15.87:0/2992029654 >> v1:172.21.15.87:6789/0 conn(0x7f9504033090 legacy=0x7f95040cf1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f9509ffb640 1 -- 172.21.15.87:0/2992029654 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950402c470 con 0x7f9504006a80 2024-04-26T02:21:32.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f9509ffb640 1 -- 172.21.15.87:0/2992029654 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f950c050460 con 0x7f9504006a80 2024-04-26T02:21:32.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f3f71fee640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f6402beb0 con 0x7f3f640cc0c0 2024-04-26T02:21:32.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f3f71fee640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f6402c120 con 0x7f3f64006e50 2024-04-26T02:21:32.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f3f71fee640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f6402c400 con 0x7f3f640cbc10 2024-04-26T02:21:32.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f3f6bfff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f3f640cc0c0 0x7f3f640d7c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53794/0 (socket says 172.21.15.87:53794) 2024-04-26T02:21:32.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f3f6bfff640 1 -- 172.21.15.87:0/2955856371 learned_addr learned my addr 172.21.15.87:0/2955856371 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f9a65098640 1 -- 172.21.15.87:0/3832458778 shutdown_connections 2024-04-26T02:21:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f9509ffb640 1 -- 172.21.15.87:0/2992029654 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f950c04fa40 con 0x7f9504006a80 2024-04-26T02:21:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f9509ffb640 1 -- 172.21.15.87:0/2992029654 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f950c04f4d0 con 0x7f9504006a80 2024-04-26T02:21:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6aa2ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f6a9c034850 0x7f6a9c15eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.87:35556/0 (socket says 172.21.15.87:35556) 2024-04-26T02:21:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6aa2ffd640 1 -- 172.21.15.87:0/1379268622 learned_addr learned my addr 172.21.15.87:0/1379268622 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6aa3fff640 1 -- 172.21.15.87:0/1379268622 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a9c02d7b0 con 0x7f6a9c0cc480 2024-04-26T02:21:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f950bfff640 1 -- 172.21.15.87:0/2992029654 >> v1:172.21.15.87:6790/0 conn(0x7f9504006a80 legacy=0x7f95040c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6aa3fff640 1 -- 172.21.15.87:0/1379268622 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a9c02da90 con 0x7f6a9c034850 2024-04-26T02:21:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.845+0000 7f9a65098640 1 -- 172.21.15.87:0/3832458778 wait complete. 2024-04-26T02:21:32.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f9a65098640 1 Processor -- start 2024-04-26T02:21:32.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2121936105 0 0) 0x7f6a9c02d7b0 con 0x7f6a9c0cc480 2024-04-26T02:21:32.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6aa3fff640 1 -- 172.21.15.87:0/1379268622 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a9c15f1c0 con 0x7f6a9c0c9450 2024-04-26T02:21:32.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a8c003170 con 0x7f6a9c0cc480 2024-04-26T02:21:32.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2322360575 0 0) 0x7f6a9c15f1c0 con 0x7f6a9c0c9450 2024-04-26T02:21:32.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a9c02d7b0 con 0x7f6a9c0c9450 2024-04-26T02:21:32.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3500645614 0 0) 0x7f6a9c02da90 con 0x7f6a9c034850 2024-04-26T02:21:32.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a9c15f1c0 con 0x7f6a9c034850 2024-04-26T02:21:32.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1015533329 0 0) 0x7f6a8c003170 con 0x7f6a9c0cc480 2024-04-26T02:21:32.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a9c02da90 con 0x7f6a9c0cc480 2024-04-26T02:21:32.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f6a940019b0 con 0x7f6a9c0cc480 2024-04-26T02:21:32.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6aa27fc640 1 -- 172.21.15.87:0/1379268622 >> v1:172.21.15.87:6790/0 conn(0x7f6a9c0c9450 legacy=0x7f6a9c0d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T02:21:32.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6aa27fc640 1 -- 172.21.15.87:0/1379268622 >> v1:172.21.15.87:6790/0 conn(0x7f6a9c0c9450 legacy=0x7f6a9c0d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T02:21:32.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6aa27fc640 1 --1- 172.21.15.87:0/1379268622 >> v1:172.21.15.87:6790/0 conn(0x7f6a9c0c9450 0x7f6a9c0d73f0 :-1 s=OPENED pgs=103 cs=1 l=1).handle_message read tag failed 2024-04-26T02:21:32.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6aa27fc640 1 --1- 172.21.15.87:0/1379268622 >> v1:172.21.15.87:6790/0 conn(0x7f6a9c0c9450 0x7f6a9c0d73f0 :-1 s=OPENED pgs=103 cs=1 l=1).fault on lossy channel, failing 2024-04-26T02:21:32.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2633864006 0 0) 0x7f6a9c02da90 con 0x7f6a9c0cc480 2024-04-26T02:21:32.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 >> v1:172.21.15.87:6790/0 conn(0x7f6a9c0c9450 legacy=0x7f6a9c0d73f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T02:21:32.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 >> v1:172.21.15.122:6789/0 conn(0x7f6a9c034850 legacy=0x7f6a9c15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9c16aa90 con 0x7f6a9c0cc480 2024-04-26T02:21:32.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6aa3fff640 1 -- 172.21.15.87:0/1379268622 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6a9c16acc0 con 0x7f6a9c0cc480 2024-04-26T02:21:32.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6aa3fff640 1 -- 172.21.15.87:0/1379268622 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6a9c0c4ea0 con 0x7f6a9c0cc480 2024-04-26T02:21:32.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a94003620 con 0x7f6a9c0cc480 2024-04-26T02:21:32.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f6a94003970 con 0x7f6a9c0cc480 2024-04-26T02:21:32.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f9a65098640 1 -- start start 2024-04-26T02:21:32.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f9a65098640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a601441d0 con 0x7f9a600b0c20 2024-04-26T02:21:32.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f9a65098640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a601dc160 con 0x7f9a600b3660 2024-04-26T02:21:32.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f9a65098640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a601db3f0 con 0x7f9a6006b410 2024-04-26T02:21:32.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f9a5f7fe640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f9a600b0c20 0x7f9a601deeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53812/0 (socket says 172.21.15.87:53812) 2024-04-26T02:21:32.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f9a5f7fe640 1 -- 172.21.15.87:0/3123460239 learned_addr learned my addr 172.21.15.87:0/3123460239 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6aa3fff640 1 -- 172.21.15.87:0/1379268622 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a9c004ce0 con 0x7f6a9c0cc480 2024-04-26T02:21:32.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.853+0000 7f950bfff640 1 -- 172.21.15.87:0/2992029654 shutdown_connections 2024-04-26T02:21:32.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.853+0000 7f950bfff640 1 -- 172.21.15.87:0/2992029654 >> 172.21.15.87:0/2992029654 conn(0x7f9504005900 msgr2=0x7f9504005d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.849+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f6a94003b10 con 0x7f6a9c0cc480 2024-04-26T02:21:32.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.853+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7f6a9407fdc0 con 0x7f6a9c0cc480 2024-04-26T02:21:32.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.853+0000 7f950bfff640 1 -- 172.21.15.87:0/2992029654 shutdown_connections 2024-04-26T02:21:32.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.853+0000 7f950bfff640 1 -- 172.21.15.87:0/2992029654 wait complete. 2024-04-26T02:21:32.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.853+0000 7f950bfff640 1 Processor -- start 2024-04-26T02:21:32.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.853+0000 7f950bfff640 1 -- start start 2024-04-26T02:21:32.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.853+0000 7f950bfff640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95040cad90 con 0x7f9504006a80 2024-04-26T02:21:32.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.853+0000 7f950bfff640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95040cb000 con 0x7f95040cf8e0 2024-04-26T02:21:32.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.853+0000 7f950bfff640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95040cb2e0 con 0x7f9504033090 2024-04-26T02:21:32.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.853+0000 7f950a7fc640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f9504033090 0x7f95040c9c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40420/0 (socket says 172.21.15.87:40420) 2024-04-26T02:21:32.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.853+0000 7f950a7fc640 1 -- 172.21.15.87:0/2764880625 learned_addr learned my addr 172.21.15.87:0/2764880625 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.853+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1524598944 0 0) 0x7f95040cb2e0 con 0x7f9504033090 2024-04-26T02:21:32.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.853+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94f0003530 con 0x7f9504033090 2024-04-26T02:21:32.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2276157628 0 0) 0x7f95040cb000 con 0x7f95040cf8e0 2024-04-26T02:21:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f6a7fc640 1 -- 172.21.15.87:0/2955856371 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1458507731 0 0) 0x7f3f6402c400 con 0x7f3f640cbc10 2024-04-26T02:21:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f6a7fc640 1 -- 172.21.15.87:0/2955856371 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f5c003530 con 0x7f3f640cbc10 2024-04-26T02:21:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95040cb2e0 con 0x7f95040cf8e0 2024-04-26T02:21:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f6a7fc640 1 -- 172.21.15.87:0/2955856371 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 824988150 0 0) 0x7f3f6402beb0 con 0x7f3f640cc0c0 2024-04-26T02:21:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 861780256 0 0) 0x7f94f0003530 con 0x7f9504033090 2024-04-26T02:21:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f6a7fc640 1 -- 172.21.15.87:0/2955856371 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f6402c400 con 0x7f3f640cc0c0 2024-04-26T02:21:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f95040cb000 con 0x7f9504033090 2024-04-26T02:21:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9500001990 con 0x7f9504033090 2024-04-26T02:21:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f6a7fc640 1 -- 172.21.15.87:0/2955856371 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 130708796 0 0) 0x7f3f6402c120 con 0x7f3f64006e50 2024-04-26T02:21:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f6a7fc640 1 -- 172.21.15.87:0/2955856371 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f6402beb0 con 0x7f3f64006e50 2024-04-26T02:21:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f6a7fc640 1 -- 172.21.15.87:0/2955856371 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3082087987 0 0) 0x7f3f5c003530 con 0x7f3f640cbc10 2024-04-26T02:21:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f6a7fc640 1 -- 172.21.15.87:0/2955856371 >> v1:172.21.15.122:6789/0 conn(0x7f3f64006e50 legacy=0x7f3f640c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f6a7fc640 1 -- 172.21.15.87:0/2955856371 >> v1:172.21.15.87:6789/0 conn(0x7f3f640cc0c0 legacy=0x7f3f640d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2336304003 0 0) 0x7f95040cb000 con 0x7f9504033090 2024-04-26T02:21:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f6a7fc640 1 -- 172.21.15.87:0/2955856371 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f640c9750 con 0x7f3f640cbc10 2024-04-26T02:21:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f6a7fc640 1 -- 172.21.15.87:0/2955856371 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3f600014d0 con 0x7f3f640cbc10 2024-04-26T02:21:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 >> v1:172.21.15.122:6789/0 conn(0x7f95040cf8e0 legacy=0x7f9504163240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 >> v1:172.21.15.87:6789/0 conn(0x7f9504006a80 legacy=0x7f95040cb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95040ce4e0 con 0x7f9504033090 2024-04-26T02:21:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f950bfff640 1 -- 172.21.15.87:0/2764880625 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f95040ceda0 con 0x7f9504033090 2024-04-26T02:21:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f950bfff640 1 -- 172.21.15.87:0/2764880625 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9504034960 con 0x7f9504033090 2024-04-26T02:21:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f6a7fc640 1 -- 172.21.15.87:0/2955856371 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f60001730 con 0x7f3f640cbc10 2024-04-26T02:21:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f71fee640 1 -- 172.21.15.87:0/2955856371 >> v1:172.21.15.87:6790/0 conn(0x7f3f640cbc10 legacy=0x7f3f640d4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f950bfff640 1 -- 172.21.15.87:0/2764880625 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9504004ce0 con 0x7f9504033090 2024-04-26T02:21:32.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f71fee640 1 -- 172.21.15.87:0/2955856371 shutdown_connections 2024-04-26T02:21:32.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f71fee640 1 -- 172.21.15.87:0/2955856371 >> 172.21.15.87:0/2955856371 conn(0x7f3f64005c70 msgr2=0x7f3f64006080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f71fee640 1 -- 172.21.15.87:0/2955856371 shutdown_connections 2024-04-26T02:21:32.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9500001c90 con 0x7f9504033090 2024-04-26T02:21:32.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9500003a70 con 0x7f9504033090 2024-04-26T02:21:32.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.857+0000 7f3f71fee640 1 -- 172.21.15.87:0/2955856371 wait complete. 2024-04-26T02:21:32.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f71fee640 1 Processor -- start 2024-04-26T02:21:32.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f6a9c02c400 con 0x7f6a9c0cc480 2024-04-26T02:21:32.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f9500003c10 con 0x7f9504033090 2024-04-26T02:21:32.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f71fee640 1 -- start start 2024-04-26T02:21:32.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7f9500081170 con 0x7f9504033090 2024-04-26T02:21:32.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f71fee640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f64037220 con 0x7f3f640cd440 2024-04-26T02:21:32.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f71fee640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f640cafa0 con 0x7f3f64006e50 2024-04-26T02:21:32.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f71fee640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f640cb170 con 0x7f3f640cc0c0 2024-04-26T02:21:32.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f6b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3f64006e50 0x7f3f640cf2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.87:35578/0 (socket says 172.21.15.87:35578) 2024-04-26T02:21:32.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f6b7fe640 1 -- 172.21.15.87:0/617628359 learned_addr learned my addr 172.21.15.87:0/617628359 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f6bfff640 1 --1- 172.21.15.87:0/617628359 >> v1:172.21.15.87:6789/0 conn(0x7f3f640cd440 0x7f3f6416be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53830/0 (socket says 172.21.15.87:53830) 2024-04-26T02:21:32.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1405618038 0 0) 0x7f3f64037220 con 0x7f3f640cd440 2024-04-26T02:21:32.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f50003530 con 0x7f3f640cd440 2024-04-26T02:21:32.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 975173251 0 0) 0x7f3f640cafa0 con 0x7f3f64006e50 2024-04-26T02:21:32.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f64037220 con 0x7f3f64006e50 2024-04-26T02:21:32.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 468521118 0 0) 0x7f3f50003530 con 0x7f3f640cd440 2024-04-26T02:21:32.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f640cafa0 con 0x7f3f640cd440 2024-04-26T02:21:32.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3f6c04fa40 con 0x7f3f640cd440 2024-04-26T02:21:32.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1710034339 0 0) 0x7f3f640cafa0 con 0x7f3f640cd440 2024-04-26T02:21:32.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 >> v1:172.21.15.87:6790/0 conn(0x7f3f640cc0c0 legacy=0x7f3f640cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 >> v1:172.21.15.122:6789/0 conn(0x7f3f64006e50 legacy=0x7f3f640cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.861+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f640cb340 con 0x7f3f640cd440 2024-04-26T02:21:32.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.865+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3225821304 0 0) 0x7f9a601441d0 con 0x7f9a600b0c20 2024-04-26T02:21:32.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.865+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a4c003530 con 0x7f9a600b0c20 2024-04-26T02:21:32.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.865+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3380395033 0 0) 0x7f9a601dc160 con 0x7f9a600b3660 2024-04-26T02:21:32.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.865+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a601441d0 con 0x7f9a600b3660 2024-04-26T02:21:32.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.865+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f6c050460 con 0x7f3f640cd440 2024-04-26T02:21:32.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.865+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4221989597 0 0) 0x7f9a601db3f0 con 0x7f9a6006b410 2024-04-26T02:21:32.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.865+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a601dc160 con 0x7f9a6006b410 2024-04-26T02:21:32.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.865+0000 7f3f71fee640 1 -- 172.21.15.87:0/617628359 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3f640cd990 con 0x7f3f640cd440 2024-04-26T02:21:32.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.865+0000 7f3f71fee640 1 -- 172.21.15.87:0/617628359 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3f640cde50 con 0x7f3f640cd440 2024-04-26T02:21:32.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.865+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3f6c0689b0 con 0x7f3f640cd440 2024-04-26T02:21:32.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.865+0000 7f3f71fee640 1 -- 172.21.15.87:0/617628359 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f4c005090 con 0x7f3f640cd440 2024-04-26T02:21:32.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.865+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f3f6c068b50 con 0x7f3f640cd440 2024-04-26T02:21:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.865+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7f3f6c0fb8d0 con 0x7f3f640cd440 2024-04-26T02:21:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f9500052190 con 0x7f9504033090 2024-04-26T02:21:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3435928738 0 0) 0x7f9a4c003530 con 0x7f9a600b0c20 2024-04-26T02:21:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a601db3f0 con 0x7f9a600b0c20 2024-04-26T02:21:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2237982529 0 0) 0x7f9a601441d0 con 0x7f9a600b3660 2024-04-26T02:21:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a4c003530 con 0x7f9a600b3660 2024-04-26T02:21:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1405151992 0 0) 0x7f9a601dc160 con 0x7f9a6006b410 2024-04-26T02:21:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a601441d0 con 0x7f9a6006b410 2024-04-26T02:21:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9a50001900 con 0x7f9a600b0c20 2024-04-26T02:21:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9a54001b50 con 0x7f9a600b3660 2024-04-26T02:21:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9a580019f0 con 0x7f9a6006b410 2024-04-26T02:21:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3012150885 0 0) 0x7f9a601441d0 con 0x7f9a6006b410 2024-04-26T02:21:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 >> v1:172.21.15.122:6789/0 conn(0x7f9a600b3660 legacy=0x7f9a600b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 >> v1:172.21.15.87:6789/0 conn(0x7f9a600b0c20 legacy=0x7f9a601deeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a601db5c0 con 0x7f9a6006b410 2024-04-26T02:21:32.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a65098640 1 -- 172.21.15.87:0/3123460239 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9a600b2ad0 con 0x7f9a6006b410 2024-04-26T02:21:32.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a65098640 1 -- 172.21.15.87:0/3123460239 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9a601df790 con 0x7f9a6006b410 2024-04-26T02:21:32.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9a58002040 con 0x7f9a6006b410 2024-04-26T02:21:32.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9a58003980 con 0x7f9a6006b410 2024-04-26T02:21:32.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f3f640cde50 con 0x7f3f640cd440 2024-04-26T02:21:32.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a65098640 1 -- 172.21.15.87:0/3123460239 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a40005090 con 0x7f9a6006b410 2024-04-26T02:21:32.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.873+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f9a58018f00 con 0x7f9a6006b410 2024-04-26T02:21:32.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.877+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7f9a5807fde0 con 0x7f9a6006b410 2024-04-26T02:21:32.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.885+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f9a58050e00 con 0x7f9a6006b410 2024-04-26T02:21:32.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.913+0000 7fdeaabe8640 1 Processor -- start 2024-04-26T02:21:32.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.913+0000 7fdeaabe8640 1 -- start start 2024-04-26T02:21:32.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.913+0000 7fdea8984640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fde9c037190 0x7fde9c0c1670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40432/0 (socket says 172.21.15.87:40432) 2024-04-26T02:21:32.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.913+0000 7fdea9185640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fde9c02ad70 0x7fde9c0cfde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53832/0 (socket says 172.21.15.87:53832) 2024-04-26T02:21:32.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.913+0000 7fdea8984640 1 -- 172.21.15.87:0/1871180964 learned_addr learned my addr 172.21.15.87:0/1871180964 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdeaabe8640 1 -- 172.21.15.87:0/1871180964 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde9c02d580 con 0x7fde9c02ad70 2024-04-26T02:21:32.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdeaabe8640 1 -- 172.21.15.87:0/1871180964 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde9c02b320 con 0x7fde9c02c240 2024-04-26T02:21:32.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdeaabe8640 1 -- 172.21.15.87:0/1871180964 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde9c0c1d80 con 0x7fde9c037190 2024-04-26T02:21:32.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdea37fe640 1 -- 172.21.15.87:0/1871180964 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 753461461 0 0) 0x7fde9c0c1d80 con 0x7fde9c037190 2024-04-26T02:21:32.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdea37fe640 1 -- 172.21.15.87:0/1871180964 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde94003530 con 0x7fde9c037190 2024-04-26T02:21:32.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdea37fe640 1 -- 172.21.15.87:0/1871180964 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4105028424 0 0) 0x7fde9c02d580 con 0x7fde9c02ad70 2024-04-26T02:21:32.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdea37fe640 1 -- 172.21.15.87:0/1871180964 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde940048f0 con 0x7fde9c02ad70 2024-04-26T02:21:32.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdea37fe640 1 -- 172.21.15.87:0/1871180964 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2712747987 0 0) 0x7fde9c02b320 con 0x7fde9c02c240 2024-04-26T02:21:32.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdea37fe640 1 -- 172.21.15.87:0/1871180964 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde9c02d580 con 0x7fde9c02c240 2024-04-26T02:21:32.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdea37fe640 1 -- 172.21.15.87:0/1871180964 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1280863262 0 0) 0x7fde94003530 con 0x7fde9c037190 2024-04-26T02:21:32.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdea37fe640 1 -- 172.21.15.87:0/1871180964 >> v1:172.21.15.122:6789/0 conn(0x7fde9c02c240 legacy=0x7fde9c0c7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdea37fe640 1 -- 172.21.15.87:0/1871180964 >> v1:172.21.15.87:6789/0 conn(0x7fde9c02ad70 legacy=0x7fde9c0cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdea37fe640 1 -- 172.21.15.87:0/1871180964 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9c0c1f50 con 0x7fde9c037190 2024-04-26T02:21:32.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdea37fe640 1 -- 172.21.15.87:0/1871180964 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fde98001600 con 0x7fde9c037190 2024-04-26T02:21:32.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdea37fe640 1 -- 172.21.15.87:0/1871180964 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fde98001920 con 0x7fde9c037190 2024-04-26T02:21:32.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdea37fe640 1 -- 172.21.15.87:0/1871180964 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fde98001460 con 0x7fde9c037190 2024-04-26T02:21:32.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdeaabe8640 1 -- 172.21.15.87:0/1871180964 >> v1:172.21.15.87:6790/0 conn(0x7fde9c037190 legacy=0x7fde9c0c1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdeaabe8640 1 -- 172.21.15.87:0/1871180964 shutdown_connections 2024-04-26T02:21:32.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdeaabe8640 1 -- 172.21.15.87:0/1871180964 >> 172.21.15.87:0/1871180964 conn(0x7fde9c03a560 msgr2=0x7fde9c036820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdeaabe8640 1 -- 172.21.15.87:0/1871180964 shutdown_connections 2024-04-26T02:21:32.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdeaabe8640 1 -- 172.21.15.87:0/1871180964 wait complete. 2024-04-26T02:21:32.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdeaabe8640 1 Processor -- start 2024-04-26T02:21:32.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.921+0000 7fdeaabe8640 1 -- start start 2024-04-26T02:21:32.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdeaabe8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde9c158470 con 0x7fde9c02c240 2024-04-26T02:21:32.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdeaabe8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde9c02e410 con 0x7fde9c02ad70 2024-04-26T02:21:32.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdeaabe8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde9c0c4610 con 0x7fde9c037190 2024-04-26T02:21:32.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea8984640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fde9c02ad70 0x7fde9c02dd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.87:35588/0 (socket says 172.21.15.87:35588) 2024-04-26T02:21:32.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea9185640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fde9c037190 0x7fde9c0baec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40434/0 (socket says 172.21.15.87:40434) 2024-04-26T02:21:32.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea8984640 1 -- 172.21.15.87:0/3465100035 learned_addr learned my addr 172.21.15.87:0/3465100035 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1209831746 0 0) 0x7fde9c0c4610 con 0x7fde9c037190 2024-04-26T02:21:32.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde88003530 con 0x7fde9c037190 2024-04-26T02:21:32.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3586416416 0 0) 0x7fde9c158470 con 0x7fde9c02c240 2024-04-26T02:21:32.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde9c0c4610 con 0x7fde9c02c240 2024-04-26T02:21:32.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2672055169 0 0) 0x7fde9c02e410 con 0x7fde9c02ad70 2024-04-26T02:21:32.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde9c158470 con 0x7fde9c02ad70 2024-04-26T02:21:32.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1651471748 0 0) 0x7fde88003530 con 0x7fde9c037190 2024-04-26T02:21:32.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde9c02e410 con 0x7fde9c037190 2024-04-26T02:21:32.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fdea4050460 con 0x7fde9c037190 2024-04-26T02:21:32.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 438123584 0 0) 0x7fde9c0c4610 con 0x7fde9c02c240 2024-04-26T02:21:32.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde88003530 con 0x7fde9c02c240 2024-04-26T02:21:32.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fde90001880 con 0x7fde9c02c240 2024-04-26T02:21:32.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 404464037 0 0) 0x7fde9c158470 con 0x7fde9c02ad70 2024-04-26T02:21:32.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde9c0c4610 con 0x7fde9c02ad70 2024-04-26T02:21:32.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fde98001600 con 0x7fde9c02ad70 2024-04-26T02:21:32.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4109755423 0 0) 0x7fde9c02e410 con 0x7fde9c037190 2024-04-26T02:21:32.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 >> v1:172.21.15.122:6789/0 conn(0x7fde9c02ad70 legacy=0x7fde9c02dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 >> v1:172.21.15.87:6789/0 conn(0x7fde9c02c240 legacy=0x7fde9c158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9c159060 con 0x7fde9c037190 2024-04-26T02:21:32.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdeaabe8640 1 -- 172.21.15.87:0/3465100035 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fde9c0bb5d0 con 0x7fde9c037190 2024-04-26T02:21:32.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7fdeaabe8640 1 -- 172.21.15.87:0/3465100035 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fde9c0c7040 con 0x7fde9c037190 2024-04-26T02:21:32.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7f8e6aea6640 1 Processor -- start 2024-04-26T02:21:32.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7f8e6aea6640 1 -- start start 2024-04-26T02:21:32.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7f8e6aea6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e5c02e310 con 0x7f8e5c02cc70 2024-04-26T02:21:32.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7f8e6aea6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e5c02e5f0 con 0x7f8e5c0c2e20 2024-04-26T02:21:32.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7f8e6aea6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e5c0331d0 con 0x7f8e5c034850 2024-04-26T02:21:32.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7f8e696a3640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8e5c0c2e20 0x7f8e5c0c8bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.87:35602/0 (socket says 172.21.15.87:35602) 2024-04-26T02:21:32.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e696a3640 1 -- 172.21.15.87:0/2670468663 learned_addr learned my addr 172.21.15.87:0/2670468663 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdea404fa40 con 0x7fde9c037190 2024-04-26T02:21:32.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fdea4068640 con 0x7fde9c037190 2024-04-26T02:21:32.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7fdea40689b0 con 0x7fde9c037190 2024-04-26T02:21:32.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7f2b96be5640 1 Processor -- start 2024-04-26T02:21:32.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7f2b96be5640 1 -- start start 2024-04-26T02:21:32.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e68ea2640 1 -- 172.21.15.87:0/2670468663 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1721488647 0 0) 0x7f8e5c02e310 con 0x7f8e5c02cc70 2024-04-26T02:21:32.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7f2b96be5640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b8802a5d0 con 0x7f2b880e2d80 2024-04-26T02:21:32.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e68ea2640 1 -- 172.21.15.87:0/2670468663 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e50003530 con 0x7f8e5c02cc70 2024-04-26T02:21:32.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e68ea2640 1 -- 172.21.15.87:0/2670468663 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2361491897 0 0) 0x7f8e5c02e5f0 con 0x7f8e5c0c2e20 2024-04-26T02:21:32.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e68ea2640 1 -- 172.21.15.87:0/2670468663 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e5c02e310 con 0x7f8e5c0c2e20 2024-04-26T02:21:32.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e68ea2640 1 -- 172.21.15.87:0/2670468663 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 834508970 0 0) 0x7f8e5c02e310 con 0x7f8e5c0c2e20 2024-04-26T02:21:32.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e68ea2640 1 -- 172.21.15.87:0/2670468663 >> v1:172.21.15.87:6790/0 conn(0x7f8e5c034850 legacy=0x7f8e5c0c2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7fdea40ca4e0 con 0x7fde9c037190 2024-04-26T02:21:32.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e68ea2640 1 -- 172.21.15.87:0/2670468663 >> v1:172.21.15.87:6789/0 conn(0x7f8e5c02cc70 legacy=0x7f8e5c0c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e68ea2640 1 -- 172.21.15.87:0/2670468663 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e5c02bec0 con 0x7f8e5c0c2e20 2024-04-26T02:21:32.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e68ea2640 1 -- 172.21.15.87:0/2670468663 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8e600015a0 con 0x7f8e5c0c2e20 2024-04-26T02:21:32.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e68ea2640 1 -- 172.21.15.87:0/2670468663 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e60001800 con 0x7f8e5c0c2e20 2024-04-26T02:21:32.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e68ea2640 1 -- 172.21.15.87:0/2670468663 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8e60001bd0 con 0x7f8e5c0c2e20 2024-04-26T02:21:32.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e6aea6640 1 -- 172.21.15.87:0/2670468663 >> v1:172.21.15.122:6789/0 conn(0x7f8e5c0c2e20 legacy=0x7f8e5c0c8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e6aea6640 1 -- 172.21.15.87:0/2670468663 shutdown_connections 2024-04-26T02:21:32.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e6aea6640 1 -- 172.21.15.87:0/2670468663 >> 172.21.15.87:0/2670468663 conn(0x7f8e5c005900 msgr2=0x7f8e5c005d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e6aea6640 1 -- 172.21.15.87:0/2670468663 shutdown_connections 2024-04-26T02:21:32.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7fc0a0fb3640 1 Processor -- start 2024-04-26T02:21:32.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7fc0a0fb3640 1 -- start start 2024-04-26T02:21:32.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7fc0a0fb3640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc09c070e60 con 0x7fc09c072b60 2024-04-26T02:21:32.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7fc0a0fb3640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc09c143e10 con 0x7fc09c0703d0 2024-04-26T02:21:32.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7fc0a0fb3640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc09c0adcb0 con 0x7fc09c0af380 2024-04-26T02:21:32.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7fc09affd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc09c0703d0 0x7fc09c072450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.87:35618/0 (socket says 172.21.15.87:35618) 2024-04-26T02:21:32.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7fc09affd640 1 -- 172.21.15.87:0/2720445730 learned_addr learned my addr 172.21.15.87:0/2720445730 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7fdeaabe8640 1 -- 172.21.15.87:0/3465100035 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde84005090 con 0x7fde9c037190 2024-04-26T02:21:32.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f8e6aea6640 1 -- 172.21.15.87:0/2670468663 wait complete. 2024-04-26T02:21:32.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e6aea6640 1 Processor -- start 2024-04-26T02:21:32.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7f2b96be5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b8802c430 con 0x7f2b88006ae0 2024-04-26T02:21:32.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.925+0000 7f2b96be5640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b8802a0b0 con 0x7f2b88029c20 2024-04-26T02:21:32.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b95182640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f2b880e2d80 0x7f2b880f0c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53870/0 (socket says 172.21.15.87:53870) 2024-04-26T02:21:32.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b95182640 1 -- 172.21.15.87:0/1420244949 learned_addr learned my addr 172.21.15.87:0/1420244949 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/1420244949 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2175974621 0 0) 0x7f2b8802a5d0 con 0x7f2b880e2d80 2024-04-26T02:21:32.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e6aea6640 1 -- start start 2024-04-26T02:21:32.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e6aea6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e5c0c5200 con 0x7f8e5c034850 2024-04-26T02:21:32.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e6aea6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e5c0c3f00 con 0x7f8e5c0c2e20 2024-04-26T02:21:32.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e6aea6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e5c0c40d0 con 0x7f8e5c02cc70 2024-04-26T02:21:32.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e696a3640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f8e5c034850 0x7f8e5c0ad400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53878/0 (socket says 172.21.15.87:53878) 2024-04-26T02:21:32.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e696a3640 1 -- 172.21.15.87:0/442802684 learned_addr learned my addr 172.21.15.87:0/442802684 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2766543836 0 0) 0x7f8e5c0c5200 con 0x7f8e5c034850 2024-04-26T02:21:32.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e48003530 con 0x7f8e5c034850 2024-04-26T02:21:32.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1893092373 0 0) 0x7f8e5c0c3f00 con 0x7f8e5c0c2e20 2024-04-26T02:21:32.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e5c0c5200 con 0x7f8e5c0c2e20 2024-04-26T02:21:32.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3238222933 0 0) 0x7f8e48003530 con 0x7f8e5c034850 2024-04-26T02:21:32.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e5c0c3f00 con 0x7f8e5c034850 2024-04-26T02:21:32.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8e60001b50 con 0x7f8e5c034850 2024-04-26T02:21:32.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2876783996 0 0) 0x7f8e5c0c5200 con 0x7f8e5c0c2e20 2024-04-26T02:21:32.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e48003530 con 0x7f8e5c0c2e20 2024-04-26T02:21:32.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8e6404fa40 con 0x7f8e5c0c2e20 2024-04-26T02:21:32.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1648040457 0 0) 0x7f8e5c0c3f00 con 0x7f8e5c034850 2024-04-26T02:21:32.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 >> v1:172.21.15.87:6790/0 conn(0x7f8e5c02cc70 legacy=0x7f8e5c0b2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 >> v1:172.21.15.122:6789/0 conn(0x7f8e5c0c2e20 legacy=0x7f8e5c174330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e5c0c3470 con 0x7f8e5c034850 2024-04-26T02:21:32.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e60004b20 con 0x7f8e5c034850 2024-04-26T02:21:32.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8e60004ef0 con 0x7f8e5c034850 2024-04-26T02:21:32.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/1420244949 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b80003530 con 0x7f2b880e2d80 2024-04-26T02:21:32.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/1420244949 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 512335784 0 0) 0x7f2b8802c430 con 0x7f2b88006ae0 2024-04-26T02:21:32.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/1420244949 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b8802a5d0 con 0x7f2b88006ae0 2024-04-26T02:21:32.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/1420244949 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1328595081 0 0) 0x7f2b80003530 con 0x7f2b880e2d80 2024-04-26T02:21:32.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/1420244949 >> v1:172.21.15.87:6790/0 conn(0x7f2b88029c20 legacy=0x7f2b880e88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:21:32.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/1420244949 >> v1:172.21.15.122:6789/0 conn(0x7f2b88006ae0 legacy=0x7f2b880e2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/1420244949 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8802b0f0 con 0x7f2b880e2d80 2024-04-26T02:21:32.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/1420244949 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f2b9004fa40 con 0x7f2b880e2d80 2024-04-26T02:21:32.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/1420244949 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2b90050460 con 0x7f2b880e2d80 2024-04-26T02:21:32.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/1420244949 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f2b9004f4d0 con 0x7f2b880e2d80 2024-04-26T02:21:32.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b96be5640 1 -- 172.21.15.87:0/1420244949 >> v1:172.21.15.87:6789/0 conn(0x7f2b880e2d80 legacy=0x7f2b880f0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b96be5640 1 -- 172.21.15.87:0/1420244949 shutdown_connections 2024-04-26T02:21:32.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.929+0000 7f2b96be5640 1 -- 172.21.15.87:0/1420244949 >> 172.21.15.87:0/1420244949 conn(0x7f2b88005900 msgr2=0x7f2b88005d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f2b96be5640 1 -- 172.21.15.87:0/1420244949 shutdown_connections 2024-04-26T02:21:32.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f2b96be5640 1 -- 172.21.15.87:0/1420244949 wait complete. 2024-04-26T02:21:32.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f2b96be5640 1 Processor -- start 2024-04-26T02:21:32.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f2b96be5640 1 -- start start 2024-04-26T02:21:32.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f2b96be5640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b88030870 con 0x7f2b880e2d80 2024-04-26T02:21:32.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f2b96be5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b880cc380 con 0x7f2b88006ae0 2024-04-26T02:21:32.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f2b96be5640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b88030ae0 con 0x7f2b880e5d10 2024-04-26T02:21:32.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f2b94981640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f2b88006ae0 0x7f2b8802f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.87:35648/0 (socket says 172.21.15.87:35648) 2024-04-26T02:21:32.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.933+0000 7f2b94981640 1 -- 172.21.15.87:0/4237747582 learned_addr learned my addr 172.21.15.87:0/4237747582 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f8e6aea6640 1 -- 172.21.15.87:0/442802684 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8e5c0adb10 con 0x7f8e5c034850 2024-04-26T02:21:32.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f8e6aea6640 1 -- 172.21.15.87:0/442802684 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8e5c175a50 con 0x7f8e5c034850 2024-04-26T02:21:32.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f8e6aea6640 1 -- 172.21.15.87:0/442802684 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e5c004ce0 con 0x7f8e5c034850 2024-04-26T02:21:32.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2488411680 0 0) 0x7f2b88030870 con 0x7f2b880e2d80 2024-04-26T02:21:32.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b74003530 con 0x7f2b880e2d80 2024-04-26T02:21:32.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3860463714 0 0) 0x7f2b74003530 con 0x7f2b880e2d80 2024-04-26T02:21:32.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b88030870 con 0x7f2b880e2d80 2024-04-26T02:21:32.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f2b840014e0 con 0x7f2b880e2d80 2024-04-26T02:21:32.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 430567462 0 0) 0x7f2b880cc380 con 0x7f2b88006ae0 2024-04-26T02:21:32.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b74003530 con 0x7f2b88006ae0 2024-04-26T02:21:32.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3709997668 0 0) 0x7f2b88030870 con 0x7f2b880e2d80 2024-04-26T02:21:32.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 >> v1:172.21.15.87:6790/0 conn(0x7f2b880e5d10 legacy=0x7f2b880cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 >> v1:172.21.15.122:6789/0 conn(0x7f2b88006ae0 legacy=0x7f2b8802f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b88030dc0 con 0x7f2b880e2d80 2024-04-26T02:21:32.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2b84001bb0 con 0x7f2b880e2d80 2024-04-26T02:21:32.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f2b840039b0 con 0x7f2b880e2d80 2024-04-26T02:21:32.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b96be5640 1 -- 172.21.15.87:0/4237747582 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2b880310a0 con 0x7f2b880e2d80 2024-04-26T02:21:32.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b96be5640 1 -- 172.21.15.87:0/4237747582 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2b8802fff0 con 0x7f2b880e2d80 2024-04-26T02:21:32.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fdea40cc8e0 con 0x7fde9c037190 2024-04-26T02:21:32.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7f2b96be5640 1 -- 172.21.15.87:0/4237747582 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b8802eb00 con 0x7f2b880e2d80 2024-04-26T02:21:32.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.941+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f8e60005090 con 0x7f8e5c034850 2024-04-26T02:21:32.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.941+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7f8e60080fc0 con 0x7f8e5c034850 2024-04-26T02:21:32.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7fc09a7fc640 1 -- 172.21.15.87:0/2720445730 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1156907132 0 0) 0x7fc09c143e10 con 0x7fc09c0703d0 2024-04-26T02:21:32.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7fc09a7fc640 1 -- 172.21.15.87:0/2720445730 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc084003530 con 0x7fc09c0703d0 2024-04-26T02:21:32.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7fc09a7fc640 1 -- 172.21.15.87:0/2720445730 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2431804827 0 0) 0x7fc09c070e60 con 0x7fc09c072b60 2024-04-26T02:21:32.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7fc09a7fc640 1 -- 172.21.15.87:0/2720445730 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc09c143e10 con 0x7fc09c072b60 2024-04-26T02:21:32.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7fc09a7fc640 1 -- 172.21.15.87:0/2720445730 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 923645748 0 0) 0x7fc084003530 con 0x7fc09c0703d0 2024-04-26T02:21:32.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7fc09a7fc640 1 -- 172.21.15.87:0/2720445730 >> v1:172.21.15.87:6790/0 conn(0x7fc09c0af380 legacy=0x7fc09c143700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7fc09a7fc640 1 -- 172.21.15.87:0/2720445730 >> v1:172.21.15.87:6789/0 conn(0x7fc09c072b60 legacy=0x7fc09c151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7fc09a7fc640 1 -- 172.21.15.87:0/2720445730 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc09c0afeb0 con 0x7fc09c0703d0 2024-04-26T02:21:32.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.937+0000 7fc09a7fc640 1 -- 172.21.15.87:0/2720445730 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc0940019a0 con 0x7fc09c0703d0 2024-04-26T02:21:32.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.941+0000 7fc09a7fc640 1 -- 172.21.15.87:0/2720445730 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc094001c70 con 0x7fc09c0703d0 2024-04-26T02:21:32.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.941+0000 7fc09a7fc640 1 -- 172.21.15.87:0/2720445730 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc094002040 con 0x7fc09c0703d0 2024-04-26T02:21:32.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.941+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f2b840190d0 con 0x7f2b880e2d80 2024-04-26T02:21:32.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.941+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7f2b8407fcc0 con 0x7f2b880e2d80 2024-04-26T02:21:32.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.949+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f8e60051fe0 con 0x7f8e5c034850 2024-04-26T02:21:32.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.953+0000 7fc0a0fb3640 1 -- 172.21.15.87:0/2720445730 >> v1:172.21.15.122:6789/0 conn(0x7fc09c0703d0 legacy=0x7fc09c072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.953+0000 7fc0a0fb3640 1 -- 172.21.15.87:0/2720445730 shutdown_connections 2024-04-26T02:21:32.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.953+0000 7fc0a0fb3640 1 -- 172.21.15.87:0/2720445730 >> 172.21.15.87:0/2720445730 conn(0x7fc09c0b0210 msgr2=0x7fc09c0b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:32.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.953+0000 7fc0a0fb3640 1 -- 172.21.15.87:0/2720445730 shutdown_connections 2024-04-26T02:21:32.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.953+0000 7fc0a0fb3640 1 -- 172.21.15.87:0/2720445730 wait complete. 2024-04-26T02:21:32.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.953+0000 7fc0a0fb3640 1 Processor -- start 2024-04-26T02:21:32.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.953+0000 7fc0a0fb3640 1 -- start start 2024-04-26T02:21:32.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.953+0000 7fc0a0fb3640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc09c1dc520 con 0x7fc09c0af380 2024-04-26T02:21:32.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.953+0000 7fc0a0fb3640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc09c13d920 con 0x7fc09c072b60 2024-04-26T02:21:32.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.953+0000 7fc0a0fb3640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc09c1d81e0 con 0x7fc09c0703d0 2024-04-26T02:21:32.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc09b7fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fc09c0703d0 0x7fc09c0a84d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40500/0 (socket says 172.21.15.87:40500) 2024-04-26T02:21:32.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc09b7fe640 1 -- 172.21.15.87:0/1081323082 learned_addr learned my addr 172.21.15.87:0/1081323082 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:32.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 98317452 0 0) 0x7fc09c1d81e0 con 0x7fc09c0703d0 2024-04-26T02:21:32.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc084001ff0 con 0x7fc09c0703d0 2024-04-26T02:21:32.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 714757380 0 0) 0x7fc09c13d920 con 0x7fc09c072b60 2024-04-26T02:21:32.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc09c1d81e0 con 0x7fc09c072b60 2024-04-26T02:21:32.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3389875480 0 0) 0x7fc084001ff0 con 0x7fc09c0703d0 2024-04-26T02:21:32.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc09c13d920 con 0x7fc09c0703d0 2024-04-26T02:21:32.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc090001850 con 0x7fc09c0703d0 2024-04-26T02:21:32.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2312690979 0 0) 0x7fc09c13d920 con 0x7fc09c0703d0 2024-04-26T02:21:32.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 >> v1:172.21.15.122:6789/0 conn(0x7fc09c072b60 legacy=0x7fc09c1dbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 >> v1:172.21.15.87:6789/0 conn(0x7fc09c0af380 legacy=0x7fc09c13d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:32.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc09c1d83b0 con 0x7fc09c0703d0 2024-04-26T02:21:32.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc090003620 con 0x7fc09c0703d0 2024-04-26T02:21:32.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc090003970 con 0x7fc09c0703d0 2024-04-26T02:21:32.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f2b84050ce0 con 0x7f2b880e2d80 2024-04-26T02:21:32.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc0a0fb3640 1 -- 172.21.15.87:0/1081323082 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc09c1d88f0 con 0x7fc09c0703d0 2024-04-26T02:21:32.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.957+0000 7fc0a0fb3640 1 -- 172.21.15.87:0/1081323082 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc09c1d8df0 con 0x7fc09c0703d0 2024-04-26T02:21:32.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.961+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7fc090003b10 con 0x7fc09c0703d0 2024-04-26T02:21:32.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.961+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 4090+0+0 (unknown 1245391589 0 0) 0x7fc090091fd0 con 0x7fc09c0703d0 2024-04-26T02:21:32.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.969+0000 7fc0a0fb3640 1 -- 172.21.15.87:0/1081323082 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc07c005090 con 0x7fc09c0703d0 2024-04-26T02:21:32.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:32.977+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fc090062ff0 con 0x7fc09c0703d0 2024-04-26T02:21:33.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.169+0000 7f3f71fee640 1 -- 172.21.15.87:0/617628359 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f3f4c005380 con 0x7f3f640cd440 2024-04-26T02:21:33.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.169+0000 7f3f57fff640 1 -- 172.21.15.87:0/617628359 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 13373408) 0x7f3f6c0cc8f0 con 0x7f3f640cd440 2024-04-26T02:21:33.171 INFO:teuthology.orchestra.run.smithi087.stdout:60129542147 2024-04-26T02:21:33.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.177+0000 7f3f55ffb640 1 -- 172.21.15.87:0/617628359 >> v1:172.21.15.122:6816/416193625 conn(0x7f3f50067fc0 legacy=0x7f3f5006a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.177+0000 7f3f55ffb640 1 -- 172.21.15.87:0/617628359 >> v1:172.21.15.87:6789/0 conn(0x7f3f640cd440 legacy=0x7f3f6416be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.177+0000 7f3f55ffb640 1 -- 172.21.15.87:0/617628359 shutdown_connections 2024-04-26T02:21:33.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.177+0000 7f3f55ffb640 1 -- 172.21.15.87:0/617628359 >> 172.21.15.87:0/617628359 conn(0x7f3f64005c70 msgr2=0x7f3f64005550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:33.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.177+0000 7f3f55ffb640 1 -- 172.21.15.87:0/617628359 shutdown_connections 2024-04-26T02:21:33.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.177+0000 7f3f55ffb640 1 -- 172.21.15.87:0/617628359 wait complete. 2024-04-26T02:21:33.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.177+0000 7f6aa3fff640 1 -- 172.21.15.87:0/1379268622 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f6a9c02be40 con 0x7f6a9c0cc480 2024-04-26T02:21:33.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.177+0000 7f6a8b7fe640 1 -- 172.21.15.87:0/1379268622 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 663305998) 0x7f6a94050de0 con 0x7f6a9c0cc480 2024-04-26T02:21:33.177 INFO:teuthology.orchestra.run.smithi087.stdout:60129542145 2024-04-26T02:21:33.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.181+0000 7f6a897fa640 1 -- 172.21.15.87:0/1379268622 >> v1:172.21.15.122:6816/416193625 conn(0x7f6a8c068610 legacy=0x7f6a8c06aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.181+0000 7f6a897fa640 1 -- 172.21.15.87:0/1379268622 >> v1:172.21.15.87:6789/0 conn(0x7f6a9c0cc480 legacy=0x7f6a9c175650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.181+0000 7f6a897fa640 1 -- 172.21.15.87:0/1379268622 shutdown_connections 2024-04-26T02:21:33.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.181+0000 7f6a897fa640 1 -- 172.21.15.87:0/1379268622 >> 172.21.15.87:0/1379268622 conn(0x7f6a9c005900 msgr2=0x7f6a9c005100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:33.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.181+0000 7f6a897fa640 1 -- 172.21.15.87:0/1379268622 shutdown_connections 2024-04-26T02:21:33.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.181+0000 7f6a897fa640 1 -- 172.21.15.87:0/1379268622 wait complete. 2024-04-26T02:21:33.199 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.3 2024-04-26T02:21:33.205 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542146 got 60129542145 for osd.1 2024-04-26T02:21:33.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.209+0000 7f9a65098640 1 -- 172.21.15.87:0/3123460239 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f9a40005380 con 0x7f9a6006b410 2024-04-26T02:21:33.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.209+0000 7f9a5cff9640 1 -- 172.21.15.87:0/3123460239 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 326014871) 0x7f9a58050700 con 0x7f9a6006b410 2024-04-26T02:21:33.210 INFO:teuthology.orchestra.run.smithi087.stdout:60129542146 2024-04-26T02:21:33.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.209+0000 7f950bfff640 1 -- 172.21.15.87:0/2764880625 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f95040cef70 con 0x7f9504033090 2024-04-26T02:21:33.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.213+0000 7f94fb7fe640 1 -- 172.21.15.87:0/2764880625 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 326014871) 0x7f9500051a90 con 0x7f9504033090 2024-04-26T02:21:33.211 INFO:teuthology.orchestra.run.smithi087.stdout:60129542146 2024-04-26T02:21:33.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.213+0000 7f9a65098640 1 -- 172.21.15.87:0/3123460239 >> v1:172.21.15.122:6816/416193625 conn(0x7f9a4c0688f0 legacy=0x7f9a4c06adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.213+0000 7f9a65098640 1 -- 172.21.15.87:0/3123460239 >> v1:172.21.15.87:6790/0 conn(0x7f9a6006b410 legacy=0x7f9a601dba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.213+0000 7f9a65098640 1 -- 172.21.15.87:0/3123460239 shutdown_connections 2024-04-26T02:21:33.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.213+0000 7f9a65098640 1 -- 172.21.15.87:0/3123460239 >> 172.21.15.87:0/3123460239 conn(0x7f9a600b1220 msgr2=0x7f9a60071b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:33.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.213+0000 7f9a65098640 1 -- 172.21.15.87:0/3123460239 shutdown_connections 2024-04-26T02:21:33.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.213+0000 7f9a65098640 1 -- 172.21.15.87:0/3123460239 wait complete. 2024-04-26T02:21:33.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.213+0000 7f950bfff640 1 -- 172.21.15.87:0/2764880625 >> v1:172.21.15.122:6816/416193625 conn(0x7f94f0068040 legacy=0x7f94f006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.213+0000 7f950bfff640 1 -- 172.21.15.87:0/2764880625 >> v1:172.21.15.87:6790/0 conn(0x7f9504033090 legacy=0x7f95040c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.213+0000 7f950bfff640 1 -- 172.21.15.87:0/2764880625 shutdown_connections 2024-04-26T02:21:33.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.213+0000 7f950bfff640 1 -- 172.21.15.87:0/2764880625 >> 172.21.15.87:0/2764880625 conn(0x7f9504005900 msgr2=0x7f95040bb980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:33.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.213+0000 7f950bfff640 1 -- 172.21.15.87:0/2764880625 shutdown_connections 2024-04-26T02:21:33.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.213+0000 7f950bfff640 1 -- 172.21.15.87:0/2764880625 wait complete. 2024-04-26T02:21:33.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.225+0000 7fdeaabe8640 1 -- 172.21.15.87:0/3465100035 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fde84005380 con 0x7fde9c037190 2024-04-26T02:21:33.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.225+0000 7fdea17fa640 1 -- 172.21.15.87:0/3465100035 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (unknown 1482059429 0 663305998) 0x7fdea40cc1e0 con 0x7fde9c037190 2024-04-26T02:21:33.225 INFO:teuthology.orchestra.run.smithi087.stdout:60129542145 2024-04-26T02:21:33.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.229+0000 7fdeaabe8640 1 -- 172.21.15.87:0/3465100035 >> v1:172.21.15.122:6816/416193625 conn(0x7fde88068580 legacy=0x7fde8806aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.229+0000 7fdeaabe8640 1 -- 172.21.15.87:0/3465100035 >> v1:172.21.15.87:6790/0 conn(0x7fde9c037190 legacy=0x7fde9c0baec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.229+0000 7fdeaabe8640 1 -- 172.21.15.87:0/3465100035 shutdown_connections 2024-04-26T02:21:33.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.229+0000 7fdeaabe8640 1 -- 172.21.15.87:0/3465100035 >> 172.21.15.87:0/3465100035 conn(0x7fde9c03a560 msgr2=0x7fde9c039380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:33.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.229+0000 7fdeaabe8640 1 -- 172.21.15.87:0/3465100035 shutdown_connections 2024-04-26T02:21:33.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.229+0000 7fdeaabe8640 1 -- 172.21.15.87:0/3465100035 wait complete. 2024-04-26T02:21:33.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.233+0000 7f8e6aea6640 1 -- 172.21.15.87:0/442802684 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8e5c0329a0 con 0x7f8e5c034850 2024-04-26T02:21:33.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.237+0000 7f8e5a7fc640 1 -- 172.21.15.87:0/442802684 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 2194451396) 0x7f8e600518e0 con 0x7f8e5c034850 2024-04-26T02:21:33.233 INFO:teuthology.orchestra.run.smithi087.stdout:55834574851 2024-04-26T02:21:33.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.237+0000 7f8e6aea6640 1 -- 172.21.15.87:0/442802684 >> v1:172.21.15.122:6816/416193625 conn(0x7f8e48068310 legacy=0x7f8e4806a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.237+0000 7f8e6aea6640 1 -- 172.21.15.87:0/442802684 >> v1:172.21.15.87:6789/0 conn(0x7f8e5c034850 legacy=0x7f8e5c0ad400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.237+0000 7f8e6aea6640 1 -- 172.21.15.87:0/442802684 shutdown_connections 2024-04-26T02:21:33.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.237+0000 7f8e6aea6640 1 -- 172.21.15.87:0/442802684 >> 172.21.15.87:0/442802684 conn(0x7f8e5c005900 msgr2=0x7f8e5c0c24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:33.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.237+0000 7f8e6aea6640 1 -- 172.21.15.87:0/442802684 shutdown_connections 2024-04-26T02:21:33.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.237+0000 7f8e6aea6640 1 -- 172.21.15.87:0/442802684 wait complete. 2024-04-26T02:21:33.239 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 60129542146 for osd.0 2024-04-26T02:21:33.240 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 60129542146 for osd.5 2024-04-26T02:21:33.246 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542146 got 60129542145 for osd.7 2024-04-26T02:21:33.252 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.2 2024-04-26T02:21:33.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.257+0000 7f2b96be5640 1 -- 172.21.15.87:0/4237747582 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f2b88004ce0 con 0x7f2b880e2d80 2024-04-26T02:21:33.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.257+0000 7f2b8d7fa640 1 -- 172.21.15.87:0/4237747582 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (unknown 1274345170 0 2194451396) 0x7f2b840505e0 con 0x7f2b880e2d80 2024-04-26T02:21:33.254 INFO:teuthology.orchestra.run.smithi087.stdout:55834574851 2024-04-26T02:21:33.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.257+0000 7f2b96be5640 1 -- 172.21.15.87:0/4237747582 >> v1:172.21.15.122:6816/416193625 conn(0x7f2b74068490 legacy=0x7f2b7406a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.257+0000 7f2b96be5640 1 -- 172.21.15.87:0/4237747582 >> v1:172.21.15.87:6789/0 conn(0x7f2b880e2d80 legacy=0x7f2b880cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.257+0000 7f2b96be5640 1 -- 172.21.15.87:0/4237747582 shutdown_connections 2024-04-26T02:21:33.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.257+0000 7f2b96be5640 1 -- 172.21.15.87:0/4237747582 >> 172.21.15.87:0/4237747582 conn(0x7f2b88005900 msgr2=0x7f2b88038460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:33.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.257+0000 7f2b96be5640 1 -- 172.21.15.87:0/4237747582 shutdown_connections 2024-04-26T02:21:33.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.257+0000 7f2b96be5640 1 -- 172.21.15.87:0/4237747582 wait complete. 2024-04-26T02:21:33.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.269+0000 7fc0a0fb3640 1 -- 172.21.15.87:0/1081323082 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fc07c005380 con 0x7fc09c0703d0 2024-04-26T02:21:33.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.269+0000 7fc083fff640 1 -- 172.21.15.87:0/1081323082 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 326014871) 0x7fc0900628f0 con 0x7fc09c0703d0 2024-04-26T02:21:33.266 INFO:teuthology.orchestra.run.smithi087.stdout:60129542146 2024-04-26T02:21:33.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.269+0000 7fc0a0fb3640 1 -- 172.21.15.87:0/1081323082 >> v1:172.21.15.122:6816/416193625 conn(0x7fc084068680 legacy=0x7fc08406ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.269+0000 7fc0a0fb3640 1 -- 172.21.15.87:0/1081323082 >> v1:172.21.15.87:6790/0 conn(0x7fc09c0703d0 legacy=0x7fc09c0a84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:33.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.269+0000 7fc0a0fb3640 1 -- 172.21.15.87:0/1081323082 shutdown_connections 2024-04-26T02:21:33.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.269+0000 7fc0a0fb3640 1 -- 172.21.15.87:0/1081323082 >> 172.21.15.87:0/1081323082 conn(0x7fc09c0b0210 msgr2=0x7fc09c074e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:33.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.269+0000 7fc0a0fb3640 1 -- 172.21.15.87:0/1081323082 shutdown_connections 2024-04-26T02:21:33.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:33.269+0000 7fc0a0fb3640 1 -- 172.21.15.87:0/1081323082 wait complete. 2024-04-26T02:21:33.271 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.6 2024-04-26T02:21:33.282 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 60129542146 for osd.4 2024-04-26T02:21:34.199 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T02:21:34.205 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T02:21:34.240 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T02:21:34.241 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-26T02:21:34.246 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-26T02:21:34.254 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T02:21:34.271 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-26T02:21:34.282 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-26T02:21:34.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0aa0fd640 1 Processor -- start 2024-04-26T02:21:34.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0aa0fd640 1 -- start start 2024-04-26T02:21:34.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0aa0fd640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0a4070d40 con 0x7fe0a40af380 2024-04-26T02:21:34.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0aa0fd640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0a4072cd0 con 0x7fe0a413b4f0 2024-04-26T02:21:34.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0aa0fd640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0a4070590 con 0x7fe0a4071190 2024-04-26T02:21:34.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0a2ffd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fe0a4071190 0x7fe0a40731f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40508/0 (socket says 172.21.15.87:40508) 2024-04-26T02:21:34.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0a2ffd640 1 -- 172.21.15.87:0/1988377878 learned_addr learned my addr 172.21.15.87:0/1988377878 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0a27fc640 1 -- 172.21.15.87:0/1988377878 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2292596906 0 0) 0x7fe0a4070590 con 0x7fe0a4071190 2024-04-26T02:21:34.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0a27fc640 1 -- 172.21.15.87:0/1988377878 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe08c003530 con 0x7fe0a4071190 2024-04-26T02:21:34.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0a27fc640 1 -- 172.21.15.87:0/1988377878 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2142878703 0 0) 0x7fe0a4070d40 con 0x7fe0a40af380 2024-04-26T02:21:34.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0a27fc640 1 -- 172.21.15.87:0/1988377878 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0a4070590 con 0x7fe0a40af380 2024-04-26T02:21:34.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0a27fc640 1 -- 172.21.15.87:0/1988377878 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4024044032 0 0) 0x7fe08c003530 con 0x7fe0a4071190 2024-04-26T02:21:34.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0a27fc640 1 -- 172.21.15.87:0/1988377878 >> v1:172.21.15.122:6789/0 conn(0x7fe0a413b4f0 legacy=0x7fe0a41498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0a27fc640 1 -- 172.21.15.87:0/1988377878 >> v1:172.21.15.87:6789/0 conn(0x7fe0a40af380 legacy=0x7fe0a4141520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0a27fc640 1 -- 172.21.15.87:0/1988377878 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a40728f0 con 0x7fe0a4071190 2024-04-26T02:21:34.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0a27fc640 1 -- 172.21.15.87:0/1988377878 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fe09c0019a0 con 0x7fe0a4071190 2024-04-26T02:21:34.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0a27fc640 1 -- 172.21.15.87:0/1988377878 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe09c001c00 con 0x7fe0a4071190 2024-04-26T02:21:34.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0aa0fd640 1 -- 172.21.15.87:0/1988377878 >> v1:172.21.15.87:6790/0 conn(0x7fe0a4071190 legacy=0x7fe0a40731f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0aa0fd640 1 -- 172.21.15.87:0/1988377878 shutdown_connections 2024-04-26T02:21:34.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.361+0000 7fe0aa0fd640 1 -- 172.21.15.87:0/1988377878 >> 172.21.15.87:0/1988377878 conn(0x7fe0a40b0210 msgr2=0x7fe0a40b0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.365+0000 7fe0aa0fd640 1 -- 172.21.15.87:0/1988377878 shutdown_connections 2024-04-26T02:21:34.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.365+0000 7fe0aa0fd640 1 -- 172.21.15.87:0/1988377878 wait complete. 2024-04-26T02:21:34.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.365+0000 7fe0aa0fd640 1 Processor -- start 2024-04-26T02:21:34.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.365+0000 7fe0aa0fd640 1 -- start start 2024-04-26T02:21:34.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.365+0000 7fe0aa0fd640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0a40aa850 con 0x7fe0a40af380 2024-04-26T02:21:34.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.365+0000 7fe0aa0fd640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0a4141810 con 0x7fe0a40aad30 2024-04-26T02:21:34.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.365+0000 7fe0aa0fd640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0a40ab8f0 con 0x7fe0a413b4f0 2024-04-26T02:21:34.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.365+0000 7fe0a2ffd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fe0a413b4f0 0x7fe0a41d6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40520/0 (socket says 172.21.15.87:40520) 2024-04-26T02:21:34.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.365+0000 7fe0a37fe640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fe0a40af380 0x7fe0a41d5e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53896/0 (socket says 172.21.15.87:53896) 2024-04-26T02:21:34.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.365+0000 7fe0a2ffd640 1 -- 172.21.15.87:0/3999536291 learned_addr learned my addr 172.21.15.87:0/3999536291 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2624843676 0 0) 0x7fe0a40aa850 con 0x7fe0a40af380 2024-04-26T02:21:34.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe090003530 con 0x7fe0a40af380 2024-04-26T02:21:34.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 75941473 0 0) 0x7fe0a40ab8f0 con 0x7fe0a413b4f0 2024-04-26T02:21:34.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0a40aa850 con 0x7fe0a413b4f0 2024-04-26T02:21:34.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1174265473 0 0) 0x7fe0a4141810 con 0x7fe0a40aad30 2024-04-26T02:21:34.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0a40ab8f0 con 0x7fe0a40aad30 2024-04-26T02:21:34.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2023617448 0 0) 0x7fe090003530 con 0x7fe0a40af380 2024-04-26T02:21:34.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0a4141810 con 0x7fe0a40af380 2024-04-26T02:21:34.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fe098001a40 con 0x7fe0a40af380 2024-04-26T02:21:34.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2544142778 0 0) 0x7fe0a40aa850 con 0x7fe0a413b4f0 2024-04-26T02:21:34.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe090003530 con 0x7fe0a413b4f0 2024-04-26T02:21:34.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fe09c004b70 con 0x7fe0a413b4f0 2024-04-26T02:21:34.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe0a3fff640 1 -- 172.21.15.87:0/3999536291 >> v1:172.21.15.122:6789/0 conn(0x7fe0a40aad30 legacy=0x7fe0a40ab1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T02:21:34.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe0a3fff640 1 -- 172.21.15.87:0/3999536291 >> v1:172.21.15.122:6789/0 conn(0x7fe0a40aad30 legacy=0x7fe0a40ab1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T02:21:34.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe0a3fff640 1 --1- 172.21.15.87:0/3999536291 >> v1:172.21.15.122:6789/0 conn(0x7fe0a40aad30 0x7fe0a40ab1e0 :-1 s=OPENED pgs=117 cs=1 l=1).handle_message read tag failed 2024-04-26T02:21:34.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe0a3fff640 1 --1- 172.21.15.87:0/3999536291 >> v1:172.21.15.122:6789/0 conn(0x7fe0a40aad30 0x7fe0a40ab1e0 :-1 s=OPENED pgs=117 cs=1 l=1).fault on lossy channel, failing 2024-04-26T02:21:34.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1279790913 0 0) 0x7fe0a4141810 con 0x7fe0a40af380 2024-04-26T02:21:34.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 >> v1:172.21.15.87:6790/0 conn(0x7fe0a413b4f0 legacy=0x7fe0a41d6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 >> v1:172.21.15.122:6789/0 conn(0x7fe0a40aad30 legacy=0x7fe0a40ab1e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T02:21:34.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a41e1190 con 0x7fe0a40af380 2024-04-26T02:21:34.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe0aa0fd640 1 -- 172.21.15.87:0/3999536291 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a41de160 con 0x7fe0a40af380 2024-04-26T02:21:34.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe0aa0fd640 1 -- 172.21.15.87:0/3999536291 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe0a41de620 con 0x7fe0a40af380 2024-04-26T02:21:34.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe098001ca0 con 0x7fe0a40af380 2024-04-26T02:21:34.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fe098003a50 con 0x7fe0a40af380 2024-04-26T02:21:34.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.385+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7fe098003bf0 con 0x7fe0a40af380 2024-04-26T02:21:34.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.389+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 4090+0+0 (unknown 2211391313 0 0) 0x7fe09804fe20 con 0x7fe0a40af380 2024-04-26T02:21:34.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.389+0000 7fe0aa0fd640 1 -- 172.21.15.87:0/3999536291 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe084005090 con 0x7fe0a40af380 2024-04-26T02:21:34.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.397+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fe098052220 con 0x7fe0a40af380 2024-04-26T02:21:34.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bfb8c640 1 Processor -- start 2024-04-26T02:21:34.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bfb8c640 1 -- start start 2024-04-26T02:21:34.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bfb8c640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78b8070d40 con 0x7f78b80af380 2024-04-26T02:21:34.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bfb8c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78b8072cd0 con 0x7f78b813b4f0 2024-04-26T02:21:34.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bfb8c640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78b8070590 con 0x7f78b8071190 2024-04-26T02:21:34.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bd127640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f78b8071190 0x7f78b80731f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40528/0 (socket says 172.21.15.87:40528) 2024-04-26T02:21:34.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bd127640 1 -- 172.21.15.87:0/2951286905 learned_addr learned my addr 172.21.15.87:0/2951286905 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bc926640 1 -- 172.21.15.87:0/2951286905 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4085663367 0 0) 0x7f78b8070590 con 0x7f78b8071190 2024-04-26T02:21:34.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bc926640 1 -- 172.21.15.87:0/2951286905 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78a4003530 con 0x7f78b8071190 2024-04-26T02:21:34.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bc926640 1 -- 172.21.15.87:0/2951286905 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 776100139 0 0) 0x7f78b8070d40 con 0x7f78b80af380 2024-04-26T02:21:34.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bc926640 1 -- 172.21.15.87:0/2951286905 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78b8070590 con 0x7f78b80af380 2024-04-26T02:21:34.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bc926640 1 -- 172.21.15.87:0/2951286905 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1207109091 0 0) 0x7f78a4003530 con 0x7f78b8071190 2024-04-26T02:21:34.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bc926640 1 -- 172.21.15.87:0/2951286905 >> v1:172.21.15.122:6789/0 conn(0x7f78b813b4f0 legacy=0x7f78b81498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bc926640 1 -- 172.21.15.87:0/2951286905 >> v1:172.21.15.87:6789/0 conn(0x7f78b80af380 legacy=0x7f78b8141520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bc926640 1 -- 172.21.15.87:0/2951286905 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b80728f0 con 0x7f78b8071190 2024-04-26T02:21:34.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bc926640 1 -- 172.21.15.87:0/2951286905 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f78b00019a0 con 0x7f78b8071190 2024-04-26T02:21:34.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.413+0000 7f78bc926640 1 -- 172.21.15.87:0/2951286905 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78b0001c70 con 0x7f78b8071190 2024-04-26T02:21:34.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f78bfb8c640 1 -- 172.21.15.87:0/2951286905 >> v1:172.21.15.87:6790/0 conn(0x7f78b8071190 legacy=0x7f78b80731f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f78bfb8c640 1 -- 172.21.15.87:0/2951286905 shutdown_connections 2024-04-26T02:21:34.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f78bfb8c640 1 -- 172.21.15.87:0/2951286905 >> 172.21.15.87:0/2951286905 conn(0x7f78b80b0210 msgr2=0x7f78b80b0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f78bfb8c640 1 -- 172.21.15.87:0/2951286905 shutdown_connections 2024-04-26T02:21:34.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f78bfb8c640 1 -- 172.21.15.87:0/2951286905 wait complete. 2024-04-26T02:21:34.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d989aa640 1 Processor -- start 2024-04-26T02:21:34.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d989aa640 1 -- start start 2024-04-26T02:21:34.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d989aa640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d94071de0 con 0x7f0d941586a0 2024-04-26T02:21:34.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d989aa640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d94072050 con 0x7f0d940af380 2024-04-26T02:21:34.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d989aa640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d940b0d30 con 0x7f0d9406fb20 2024-04-26T02:21:34.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d92d9d640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f0d941586a0 0x7f0d9415aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53902/0 (socket says 172.21.15.87:53902) 2024-04-26T02:21:34.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f78bfb8c640 1 Processor -- start 2024-04-26T02:21:34.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d92d9d640 1 -- 172.21.15.87:0/485121279 learned_addr learned my addr 172.21.15.87:0/485121279 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d9159a640 1 -- 172.21.15.87:0/485121279 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 696955121 0 0) 0x7f0d94071de0 con 0x7f0d941586a0 2024-04-26T02:21:34.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d9159a640 1 -- 172.21.15.87:0/485121279 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d7c003530 con 0x7f0d941586a0 2024-04-26T02:21:34.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d9159a640 1 -- 172.21.15.87:0/485121279 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2295966203 0 0) 0x7f0d94072050 con 0x7f0d940af380 2024-04-26T02:21:34.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d9159a640 1 -- 172.21.15.87:0/485121279 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d94071de0 con 0x7f0d940af380 2024-04-26T02:21:34.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d9159a640 1 -- 172.21.15.87:0/485121279 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 697344972 0 0) 0x7f0d7c003530 con 0x7f0d941586a0 2024-04-26T02:21:34.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d9159a640 1 -- 172.21.15.87:0/485121279 >> v1:172.21.15.87:6790/0 conn(0x7f0d9406fb20 legacy=0x7f0d9406ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d9159a640 1 -- 172.21.15.87:0/485121279 >> v1:172.21.15.122:6789/0 conn(0x7f0d940af380 legacy=0x7f0d941529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d9159a640 1 -- 172.21.15.87:0/485121279 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9406f240 con 0x7f0d941586a0 2024-04-26T02:21:34.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d9159a640 1 -- 172.21.15.87:0/485121279 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f0d8c0019a0 con 0x7f0d941586a0 2024-04-26T02:21:34.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d9159a640 1 -- 172.21.15.87:0/485121279 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0d8c001c00 con 0x7f0d941586a0 2024-04-26T02:21:34.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f0d9159a640 1 -- 172.21.15.87:0/485121279 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f0d8c001fd0 con 0x7f0d941586a0 2024-04-26T02:21:34.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.417+0000 7f78bfb8c640 1 -- start start 2024-04-26T02:21:34.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f0d989aa640 1 -- 172.21.15.87:0/485121279 >> v1:172.21.15.87:6789/0 conn(0x7f0d941586a0 legacy=0x7f0d9415aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f0d989aa640 1 -- 172.21.15.87:0/485121279 shutdown_connections 2024-04-26T02:21:34.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f0d989aa640 1 -- 172.21.15.87:0/485121279 >> 172.21.15.87:0/485121279 conn(0x7f0d940b0210 msgr2=0x7f0d940b0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78bfb8c640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78b813e2f0 con 0x7f78b81def20 2024-04-26T02:21:34.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78bfb8c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78b813c420 con 0x7f78b80af380 2024-04-26T02:21:34.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78bfb8c640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78b812e4b0 con 0x7f78b813b4f0 2024-04-26T02:21:34.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f0d989aa640 1 -- 172.21.15.87:0/485121279 shutdown_connections 2024-04-26T02:21:34.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f0d989aa640 1 -- 172.21.15.87:0/485121279 wait complete. 2024-04-26T02:21:34.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78bd127640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f78b813b4f0 0x7f78b812d860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40544/0 (socket says 172.21.15.87:40544) 2024-04-26T02:21:34.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78bd127640 1 -- 172.21.15.87:0/2499741650 learned_addr learned my addr 172.21.15.87:0/2499741650 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 122150927 0 0) 0x7f78b812e4b0 con 0x7f78b813b4f0 2024-04-26T02:21:34.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f789c003530 con 0x7f78b813b4f0 2024-04-26T02:21:34.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1093663607 0 0) 0x7f78b813c420 con 0x7f78b80af380 2024-04-26T02:21:34.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78b812e4b0 con 0x7f78b80af380 2024-04-26T02:21:34.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1432585279 0 0) 0x7f78b813e2f0 con 0x7f78b81def20 2024-04-26T02:21:34.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78b813c420 con 0x7f78b81def20 2024-04-26T02:21:34.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3051866785 0 0) 0x7f789c003530 con 0x7f78b813b4f0 2024-04-26T02:21:34.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78b813e2f0 con 0x7f78b813b4f0 2024-04-26T02:21:34.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f78b0004e40 con 0x7f78b813b4f0 2024-04-26T02:21:34.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f0d989aa640 1 Processor -- start 2024-04-26T02:21:34.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f0d989aa640 1 -- start start 2024-04-26T02:21:34.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 445073175 0 0) 0x7f78b813c420 con 0x7f78b81def20 2024-04-26T02:21:34.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f789c003530 con 0x7f78b81def20 2024-04-26T02:21:34.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1977578765 0 0) 0x7f78b812e4b0 con 0x7f78b80af380 2024-04-26T02:21:34.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78b813c420 con 0x7f78b80af380 2024-04-26T02:21:34.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f78a8001610 con 0x7f78b81def20 2024-04-26T02:21:34.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f78b4001730 con 0x7f78b80af380 2024-04-26T02:21:34.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1174554128 0 0) 0x7f78b813e2f0 con 0x7f78b813b4f0 2024-04-26T02:21:34.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 >> v1:172.21.15.122:6789/0 conn(0x7f78b80af380 legacy=0x7f78b81de810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 >> v1:172.21.15.87:6789/0 conn(0x7f78b81def20 legacy=0x7f78b81e6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b813c940 con 0x7f78b813b4f0 2024-04-26T02:21:34.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f0d989aa640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d940aacc0 con 0x7f0d9406fb20 2024-04-26T02:21:34.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f0d989aa640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d941e7850 con 0x7f0d941586a0 2024-04-26T02:21:34.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f0d989aa640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d940a99c0 con 0x7f0d940af380 2024-04-26T02:21:34.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f0d9259c640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f0d9406fb20 0x7f0d940a92b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53922/0 (socket says 172.21.15.87:53922) 2024-04-26T02:21:34.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f0d9259c640 1 -- 172.21.15.87:0/3745183772 learned_addr learned my addr 172.21.15.87:0/3745183772 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3297597041 0 0) 0x7f0d940a99c0 con 0x7f0d940af380 2024-04-26T02:21:34.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78b0005350 con 0x7f78b813b4f0 2024-04-26T02:21:34.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f78b0005720 con 0x7f78b813b4f0 2024-04-26T02:21:34.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78bfb8c640 1 -- 172.21.15.87:0/2499741650 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78b812df70 con 0x7f78b813b4f0 2024-04-26T02:21:34.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f78bfb8c640 1 -- 172.21.15.87:0/2499741650 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f78b814b040 con 0x7f78b813b4f0 2024-04-26T02:21:34.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.421+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d7c003a40 con 0x7f0d940af380 2024-04-26T02:21:34.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1964894672 0 0) 0x7f0d941e7850 con 0x7f0d941586a0 2024-04-26T02:21:34.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d91d9b640 1 -- 172.21.15.87:0/3745183772 >> v1:172.21.15.87:6790/0 conn(0x7f0d940af380 legacy=0x7f0d941e7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T02:21:34.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d91d9b640 1 -- 172.21.15.87:0/3745183772 >> v1:172.21.15.87:6790/0 conn(0x7f0d940af380 legacy=0x7f0d941e7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T02:21:34.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d91d9b640 1 --1- 172.21.15.87:0/3745183772 >> v1:172.21.15.87:6790/0 conn(0x7f0d940af380 0x7f0d941e7d30 :-1 s=OPENED pgs=118 cs=1 l=1).handle_message read tag failed 2024-04-26T02:21:34.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d91d9b640 1 --1- 172.21.15.87:0/3745183772 >> v1:172.21.15.87:6790/0 conn(0x7f0d940af380 0x7f0d941e7d30 :-1 s=OPENED pgs=118 cs=1 l=1).fault on lossy channel, failing 2024-04-26T02:21:34.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d940a99c0 con 0x7f0d941586a0 2024-04-26T02:21:34.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f78bfb8c640 1 -- 172.21.15.87:0/2499741650 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78b8062e10 con 0x7f78b813b4f0 2024-04-26T02:21:34.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3295023465 0 0) 0x7f0d940aacc0 con 0x7f0d9406fb20 2024-04-26T02:21:34.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d941e7850 con 0x7f0d9406fb20 2024-04-26T02:21:34.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2786027117 0 0) 0x7f0d940a99c0 con 0x7f0d941586a0 2024-04-26T02:21:34.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d940aacc0 con 0x7f0d941586a0 2024-04-26T02:21:34.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f0d8c001e30 con 0x7f0d941586a0 2024-04-26T02:21:34.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 <== mon.1 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2885717906 0 0) 0x7f0d940aacc0 con 0x7f0d941586a0 2024-04-26T02:21:34.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 >> v1:172.21.15.87:6790/0 conn(0x7f0d940af380 legacy=0x7f0d941e7d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T02:21:34.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 >> v1:172.21.15.87:6789/0 conn(0x7f0d9406fb20 legacy=0x7f0d940a92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d941e8440 con 0x7f0d941586a0 2024-04-26T02:21:34.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d989aa640 1 -- 172.21.15.87:0/3745183772 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0d941551c0 con 0x7f0d941586a0 2024-04-26T02:21:34.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d989aa640 1 -- 172.21.15.87:0/3745183772 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0d94089f00 con 0x7f0d941586a0 2024-04-26T02:21:34.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 <== mon.1 v1:172.21.15.122:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0d8c001c00 con 0x7f0d941586a0 2024-04-26T02:21:34.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.425+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 <== mon.1 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f0d8c004e20 con 0x7f0d941586a0 2024-04-26T02:21:34.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.429+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 <== mon.1 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f0d8c01a540 con 0x7f0d941586a0 2024-04-26T02:21:34.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.429+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 <== mon.1 v1:172.21.15.122:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 4090+0+0 (unknown 2211391313 0 0) 0x7f0d8c0811b0 con 0x7f0d941586a0 2024-04-26T02:21:34.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.429+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f78b0001ea0 con 0x7f78b813b4f0 2024-04-26T02:21:34.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.429+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 4090+0+0 (unknown 2211391313 0 0) 0x7f78b0081400 con 0x7f78b813b4f0 2024-04-26T02:21:34.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.433+0000 7f0d989aa640 1 -- 172.21.15.87:0/3745183772 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d78005090 con 0x7f0d941586a0 2024-04-26T02:21:34.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.433+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f78b0052420 con 0x7f78b813b4f0 2024-04-26T02:21:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.437+0000 7fbb4b0b7640 1 Processor -- start 2024-04-26T02:21:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.437+0000 7fbb4b0b7640 1 -- start start 2024-04-26T02:21:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb4b0b7640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3c02beb0 con 0x7fbb3c0cc0c0 2024-04-26T02:21:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb4b0b7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3c02c120 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb4b0b7640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3c02c400 con 0x7fbb3c006e50 2024-04-26T02:21:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb4a0b5640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fbb3c006e50 0x7fbb3c0c8dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40572/0 (socket says 172.21.15.87:40572) 2024-04-26T02:21:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb4a0b5640 1 -- 172.21.15.87:0/1667445088 learned_addr learned my addr 172.21.15.87:0/1667445088 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb490b3640 1 -- 172.21.15.87:0/1667445088 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1723009246 0 0) 0x7fbb3c02beb0 con 0x7fbb3c0cc0c0 2024-04-26T02:21:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb490b3640 1 -- 172.21.15.87:0/1667445088 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb30003530 con 0x7fbb3c0cc0c0 2024-04-26T02:21:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb490b3640 1 -- 172.21.15.87:0/1667445088 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2095372264 0 0) 0x7fbb30003530 con 0x7fbb3c0cc0c0 2024-04-26T02:21:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb490b3640 1 -- 172.21.15.87:0/1667445088 >> v1:172.21.15.87:6790/0 conn(0x7fbb3c006e50 legacy=0x7fbb3c0c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb490b3640 1 -- 172.21.15.87:0/1667445088 >> v1:172.21.15.122:6789/0 conn(0x7fbb3c0cbc10 legacy=0x7fbb3c0d4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb490b3640 1 -- 172.21.15.87:0/1667445088 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3c0c9750 con 0x7fbb3c0cc0c0 2024-04-26T02:21:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb490b3640 1 -- 172.21.15.87:0/1667445088 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fbb44050460 con 0x7fbb3c0cc0c0 2024-04-26T02:21:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb490b3640 1 -- 172.21.15.87:0/1667445088 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbb4404fa40 con 0x7fbb3c0cc0c0 2024-04-26T02:21:34.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/1667445088 >> v1:172.21.15.87:6789/0 conn(0x7fbb3c0cc0c0 legacy=0x7fbb3c0d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/1667445088 shutdown_connections 2024-04-26T02:21:34.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/1667445088 >> 172.21.15.87:0/1667445088 conn(0x7fbb3c005c70 msgr2=0x7fbb3c006080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/1667445088 shutdown_connections 2024-04-26T02:21:34.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.441+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/1667445088 wait complete. 2024-04-26T02:21:34.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.445+0000 7fbb4b0b7640 1 Processor -- start 2024-04-26T02:21:34.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.445+0000 7fbb4b0b7640 1 -- start start 2024-04-26T02:21:34.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.445+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 <== mon.1 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f0d8c0521d0 con 0x7f0d941586a0 2024-04-26T02:21:34.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.445+0000 7fbb4a0b5640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fbb3c006e50 0x7fbb3c164500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40574/0 (socket says 172.21.15.87:40574) 2024-04-26T02:21:34.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.445+0000 7fbb498b4640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fbb3c0cbc10 0x7fbb3c0bd860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53952/0 (socket says 172.21.15.87:53952) 2024-04-26T02:21:34.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.445+0000 7fbb4a0b5640 1 -- 172.21.15.87:0/309833182 learned_addr learned my addr 172.21.15.87:0/309833182 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.449+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/309833182 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3c0be680 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.449+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/309833182 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3c0cde00 con 0x7fbb3c0cc0c0 2024-04-26T02:21:34.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.449+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/309833182 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3c0ce0e0 con 0x7fbb3c006e50 2024-04-26T02:21:34.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1009091399 0 0) 0x7fbb3c0be680 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb28003530 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 399483971 0 0) 0x7fbb3c0cde00 con 0x7fbb3c0cc0c0 2024-04-26T02:21:34.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb3c0be680 con 0x7fbb3c0cc0c0 2024-04-26T02:21:34.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2025497255 0 0) 0x7fbb3c0ce0e0 con 0x7fbb3c006e50 2024-04-26T02:21:34.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb3c0cde00 con 0x7fbb3c006e50 2024-04-26T02:21:34.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3442818560 0 0) 0x7fbb28003530 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb3c0ce0e0 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fbb400017f0 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2050045686 0 0) 0x7fbb3c0cde00 con 0x7fbb3c006e50 2024-04-26T02:21:34.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb28003530 con 0x7fbb3c006e50 2024-04-26T02:21:34.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fbb340016f0 con 0x7fbb3c006e50 2024-04-26T02:21:34.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3873834699 0 0) 0x7fbb3c0ce0e0 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 >> v1:172.21.15.87:6790/0 conn(0x7fbb3c006e50 legacy=0x7fbb3c164500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 >> v1:172.21.15.122:6789/0 conn(0x7fbb3c0cc0c0 legacy=0x7fbb3c0bdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3c0c42b0 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/309833182 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbb3c0cfe00 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/309833182 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbb3c0cd050 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/309833182 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb3c005130 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbb40003620 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fbb40003970 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7fbb40003b50 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.453+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 4090+0+0 (unknown 2211391313 0 0) 0x7fbb40080f50 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.457+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fbb40052010 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.457+0000 7fb6c3fff640 1 Processor -- start 2024-04-26T02:21:34.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c3fff640 1 -- start start 2024-04-26T02:21:34.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c3fff640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc02c630 con 0x7fb6bc02a890 2024-04-26T02:21:34.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c3fff640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc035490 con 0x7fb6bc0c11b0 2024-04-26T02:21:34.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c3fff640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc02b710 con 0x7fb6bc034850 2024-04-26T02:21:34.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c27fc640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fb6bc02a890 0x7fb6bc0c0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53958/0 (socket says 172.21.15.87:53958) 2024-04-26T02:21:34.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c27fc640 1 -- 172.21.15.87:0/4168169622 learned_addr learned my addr 172.21.15.87:0/4168169622 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c1ffb640 1 -- 172.21.15.87:0/4168169622 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3608784582 0 0) 0x7fb6bc035490 con 0x7fb6bc0c11b0 2024-04-26T02:21:34.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c1ffb640 1 -- 172.21.15.87:0/4168169622 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6ac003530 con 0x7fb6bc0c11b0 2024-04-26T02:21:34.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c1ffb640 1 -- 172.21.15.87:0/4168169622 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3705577693 0 0) 0x7fb6bc02c630 con 0x7fb6bc02a890 2024-04-26T02:21:34.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c1ffb640 1 -- 172.21.15.87:0/4168169622 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6bc035490 con 0x7fb6bc02a890 2024-04-26T02:21:34.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c1ffb640 1 -- 172.21.15.87:0/4168169622 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 17972932 0 0) 0x7fb6bc035490 con 0x7fb6bc02a890 2024-04-26T02:21:34.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c1ffb640 1 -- 172.21.15.87:0/4168169622 >> v1:172.21.15.87:6790/0 conn(0x7fb6bc034850 legacy=0x7fb6bc0c6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:21:34.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c1ffb640 1 -- 172.21.15.87:0/4168169622 >> v1:172.21.15.122:6789/0 conn(0x7fb6bc0c11b0 legacy=0x7fb6bc0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c1ffb640 1 -- 172.21.15.87:0/4168169622 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6bc02a090 con 0x7fb6bc02a890 2024-04-26T02:21:34.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c1ffb640 1 -- 172.21.15.87:0/4168169622 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb6b4001620 con 0x7fb6bc02a890 2024-04-26T02:21:34.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb6c1ffb640 1 -- 172.21.15.87:0/4168169622 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6b4001880 con 0x7fb6bc02a890 2024-04-26T02:21:34.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb6c1ffb640 1 -- 172.21.15.87:0/4168169622 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb6b4001c50 con 0x7fb6bc02a890 2024-04-26T02:21:34.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb6c3fff640 1 -- 172.21.15.87:0/4168169622 >> v1:172.21.15.87:6789/0 conn(0x7fb6bc02a890 legacy=0x7fb6bc0c0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb0eaa35640 1 Processor -- start 2024-04-26T02:21:34.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb0eaa35640 1 -- start start 2024-04-26T02:21:34.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb0eaa35640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e4072660 con 0x7fb0e4072250 2024-04-26T02:21:34.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb0eaa35640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e4153480 con 0x7fb0e406f1d0 2024-04-26T02:21:34.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb0eaa35640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e4070520 con 0x7fb0e406f5b0 2024-04-26T02:21:34.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb0e3fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb0e406f1d0 0x7fb0e4071b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.87:35772/0 (socket says 172.21.15.87:35772) 2024-04-26T02:21:34.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb0e3fff640 1 -- 172.21.15.87:0/3758059176 learned_addr learned my addr 172.21.15.87:0/3758059176 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb0e2ffd640 1 -- 172.21.15.87:0/3758059176 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2234442598 0 0) 0x7fb0e4153480 con 0x7fb0e406f1d0 2024-04-26T02:21:34.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb0e2ffd640 1 -- 172.21.15.87:0/3758059176 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0cc003530 con 0x7fb0e406f1d0 2024-04-26T02:21:34.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.461+0000 7fb0e2ffd640 1 -- 172.21.15.87:0/3758059176 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3746334486 0 0) 0x7fb0e4072660 con 0x7fb0e4072250 2024-04-26T02:21:34.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb0e2ffd640 1 -- 172.21.15.87:0/3758059176 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0e4153480 con 0x7fb0e4072250 2024-04-26T02:21:34.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb0e2ffd640 1 -- 172.21.15.87:0/3758059176 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2106874702 0 0) 0x7fb0e4070520 con 0x7fb0e406f5b0 2024-04-26T02:21:34.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb0e2ffd640 1 -- 172.21.15.87:0/3758059176 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0e4072660 con 0x7fb0e406f5b0 2024-04-26T02:21:34.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb0e2ffd640 1 -- 172.21.15.87:0/3758059176 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3816606084 0 0) 0x7fb0cc003530 con 0x7fb0e406f1d0 2024-04-26T02:21:34.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb0e2ffd640 1 -- 172.21.15.87:0/3758059176 >> v1:172.21.15.87:6790/0 conn(0x7fb0e406f5b0 legacy=0x7fb0e4152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb0e2ffd640 1 -- 172.21.15.87:0/3758059176 >> v1:172.21.15.87:6789/0 conn(0x7fb0e4072250 legacy=0x7fb0e4152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb0e2ffd640 1 -- 172.21.15.87:0/3758059176 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e40710f0 con 0x7fb0e406f1d0 2024-04-26T02:21:34.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb0e2ffd640 1 -- 172.21.15.87:0/3758059176 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb0d40015a0 con 0x7fb0e406f1d0 2024-04-26T02:21:34.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb6c3fff640 1 -- 172.21.15.87:0/4168169622 shutdown_connections 2024-04-26T02:21:34.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb6c3fff640 1 -- 172.21.15.87:0/4168169622 >> 172.21.15.87:0/4168169622 conn(0x7fb6bc005900 msgr2=0x7fb6bc005d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb6c3fff640 1 -- 172.21.15.87:0/4168169622 shutdown_connections 2024-04-26T02:21:34.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb0e2ffd640 1 -- 172.21.15.87:0/3758059176 <== mon.1 v1:172.21.15.122:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb0d4001800 con 0x7fb0e406f1d0 2024-04-26T02:21:34.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb0e2ffd640 1 -- 172.21.15.87:0/3758059176 <== mon.1 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb0d4001b90 con 0x7fb0e406f1d0 2024-04-26T02:21:34.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb0eaa35640 1 -- 172.21.15.87:0/3758059176 >> v1:172.21.15.122:6789/0 conn(0x7fb0e406f1d0 legacy=0x7fb0e4071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb0eaa35640 1 -- 172.21.15.87:0/3758059176 shutdown_connections 2024-04-26T02:21:34.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb0eaa35640 1 -- 172.21.15.87:0/3758059176 >> 172.21.15.87:0/3758059176 conn(0x7fb0e40b4230 msgr2=0x7fb0e406b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb6c3fff640 1 -- 172.21.15.87:0/4168169622 wait complete. 2024-04-26T02:21:34.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb6c3fff640 1 Processor -- start 2024-04-26T02:21:34.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb0eaa35640 1 -- 172.21.15.87:0/3758059176 shutdown_connections 2024-04-26T02:21:34.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb0eaa35640 1 -- 172.21.15.87:0/3758059176 wait complete. 2024-04-26T02:21:34.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb6c3fff640 1 -- start start 2024-04-26T02:21:34.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0eaa35640 1 Processor -- start 2024-04-26T02:21:34.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0eaa35640 1 -- start start 2024-04-26T02:21:34.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb6c3fff640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc16a5e0 con 0x7fb6bc0c11b0 2024-04-26T02:21:34.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb6c3fff640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc0c9040 con 0x7fb6bc02a890 2024-04-26T02:21:34.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb6c3fff640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc0c9320 con 0x7fb6bc034850 2024-04-26T02:21:34.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb6c37fe640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fb6bc0c11b0 0x7fb6bc169ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53976/0 (socket says 172.21.15.87:53976) 2024-04-26T02:21:34.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.465+0000 7fb6c37fe640 1 -- 172.21.15.87:0/2517792654 learned_addr learned my addr 172.21.15.87:0/2517792654 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2474628322 0 0) 0x7fb6bc16a5e0 con 0x7fb6bc0c11b0 2024-04-26T02:21:34.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6b0003530 con 0x7fb6bc0c11b0 2024-04-26T02:21:34.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3399532938 0 0) 0x7fb6bc0c9320 con 0x7fb6bc034850 2024-04-26T02:21:34.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6bc16a5e0 con 0x7fb6bc034850 2024-04-26T02:21:34.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 245897077 0 0) 0x7fb6bc0c9040 con 0x7fb6bc02a890 2024-04-26T02:21:34.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0e3fff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fb0e406f1d0 0x7fb0e41e0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53990/0 (socket says 172.21.15.87:53990) 2024-04-26T02:21:34.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6bc0c9320 con 0x7fb6bc02a890 2024-04-26T02:21:34.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0e3fff640 1 -- 172.21.15.87:0/997803590 learned_addr learned my addr 172.21.15.87:0/997803590 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 131709225 0 0) 0x7fb6bc16a5e0 con 0x7fb6bc034850 2024-04-26T02:21:34.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6bc0c9040 con 0x7fb6bc034850 2024-04-26T02:21:34.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0eaa35640 1 -- 172.21.15.87:0/997803590 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e40ac670 con 0x7fb0e406f1d0 2024-04-26T02:21:34.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1414038881 0 0) 0x7fb6b0003530 con 0x7fb6bc0c11b0 2024-04-26T02:21:34.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0eaa35640 1 -- 172.21.15.87:0/997803590 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e4156d20 con 0x7fb0e4072250 2024-04-26T02:21:34.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1470037772 0 0) 0x7fb0e40ac670 con 0x7fb0e406f1d0 2024-04-26T02:21:34.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6bc16a5e0 con 0x7fb6bc0c11b0 2024-04-26T02:21:34.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1308769940 0 0) 0x7fb6bc0c9320 con 0x7fb6bc02a890 2024-04-26T02:21:34.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6b0003530 con 0x7fb6bc02a890 2024-04-26T02:21:34.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb6b4001520 con 0x7fb6bc034850 2024-04-26T02:21:34.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb6c404fa40 con 0x7fb6bc0c11b0 2024-04-26T02:21:34.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0eaa35640 1 -- 172.21.15.87:0/997803590 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e41e7f50 con 0x7fb0e406f5b0 2024-04-26T02:21:34.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0d0003530 con 0x7fb0e406f1d0 2024-04-26T02:21:34.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4035099843 0 0) 0x7fb0e4156d20 con 0x7fb0e4072250 2024-04-26T02:21:34.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0e40ac670 con 0x7fb0e4072250 2024-04-26T02:21:34.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb6b80016b0 con 0x7fb6bc02a890 2024-04-26T02:21:34.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 156384897 0 0) 0x7fb6bc0c9040 con 0x7fb6bc034850 2024-04-26T02:21:34.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 >> v1:172.21.15.122:6789/0 conn(0x7fb6bc02a890 legacy=0x7fb6bc02e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 >> v1:172.21.15.87:6789/0 conn(0x7fb6bc0c11b0 legacy=0x7fb6bc169ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 357634121 0 0) 0x7fb0d0003530 con 0x7fb0e406f1d0 2024-04-26T02:21:34.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0e4156d20 con 0x7fb0e406f1d0 2024-04-26T02:21:34.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb0d4001620 con 0x7fb0e406f1d0 2024-04-26T02:21:34.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 684528923 0 0) 0x7fb0e41e7f50 con 0x7fb0e406f5b0 2024-04-26T02:21:34.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0d0003530 con 0x7fb0e406f5b0 2024-04-26T02:21:34.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3848286913 0 0) 0x7fb0e40ac670 con 0x7fb0e4072250 2024-04-26T02:21:34.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0e41e7f50 con 0x7fb0e4072250 2024-04-26T02:21:34.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.469+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb0dc001b60 con 0x7fb0e4072250 2024-04-26T02:21:34.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6bc0c9600 con 0x7fb6bc034850 2024-04-26T02:21:34.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 <== mon.1 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2705803555 0 0) 0x7fb0e41e7f50 con 0x7fb0e4072250 2024-04-26T02:21:34.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 >> v1:172.21.15.87:6790/0 conn(0x7fb0e406f5b0 legacy=0x7fb0e41e7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 >> v1:172.21.15.87:6789/0 conn(0x7fb0e406f1d0 legacy=0x7fb0e41e0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e40ab810 con 0x7fb0e4072250 2024-04-26T02:21:34.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb0eaa35640 1 -- 172.21.15.87:0/997803590 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e41e0880 con 0x7fb0e4072250 2024-04-26T02:21:34.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb0eaa35640 1 -- 172.21.15.87:0/997803590 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb0e41e1730 con 0x7fb0e4072250 2024-04-26T02:21:34.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb6c3fff640 1 -- 172.21.15.87:0/2517792654 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb6bc02ea30 con 0x7fb6bc034850 2024-04-26T02:21:34.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb6c3fff640 1 -- 172.21.15.87:0/2517792654 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6bc0c7c80 con 0x7fb6bc034850 2024-04-26T02:21:34.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6b4000fa0 con 0x7fb6bc034850 2024-04-26T02:21:34.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 <== mon.1 v1:172.21.15.122:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb0dc003980 con 0x7fb0e4072250 2024-04-26T02:21:34.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 <== mon.1 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb0dc003cd0 con 0x7fb0e4072250 2024-04-26T02:21:34.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb0ca7fc640 1 -- 172.21.15.87:0/997803590 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0e404fd20 con 0x7fb0e4072250 2024-04-26T02:21:34.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb37ffff640 1 Processor -- start 2024-04-26T02:21:34.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb37ffff640 1 -- start start 2024-04-26T02:21:34.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb6b4004d70 con 0x7fb6bc034850 2024-04-26T02:21:34.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7f2907fff640 1 Processor -- start 2024-04-26T02:21:34.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb6c3fff640 1 -- 172.21.15.87:0/2517792654 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a4005090 con 0x7fb6bc034850 2024-04-26T02:21:34.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb37ffff640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb380072430 con 0x7fb3800af120 2024-04-26T02:21:34.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb37ffff640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb380141490 con 0x7fb38013eb30 2024-04-26T02:21:34.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb37ffff640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb380071120 con 0x7fb3800af500 2024-04-26T02:21:34.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb37e7fc640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fb3800af500 0x7fb38013e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40638/0 (socket says 172.21.15.87:40638) 2024-04-26T02:21:34.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7fb37e7fc640 1 -- 172.21.15.87:0/1468073113 learned_addr learned my addr 172.21.15.87:0/1468073113 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7f2907fff640 1 -- start start 2024-04-26T02:21:34.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7f2907fff640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f290002c630 con 0x7f29000c11b0 2024-04-26T02:21:34.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7f2907fff640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2900035490 con 0x7f2900034850 2024-04-26T02:21:34.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.473+0000 7f2907fff640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f290002b710 con 0x7f290002a890 2024-04-26T02:21:34.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7f29067fc640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f290002a890 0x7f29000c0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40654/0 (socket says 172.21.15.87:40654) 2024-04-26T02:21:34.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7f29067fc640 1 -- 172.21.15.87:0/3535691046 learned_addr learned my addr 172.21.15.87:0/3535691046 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7fb37dffb640 1 -- 172.21.15.87:0/1468073113 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3405228692 0 0) 0x7fb380072430 con 0x7fb3800af120 2024-04-26T02:21:34.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7fb37dffb640 1 -- 172.21.15.87:0/1468073113 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb368003530 con 0x7fb3800af120 2024-04-26T02:21:34.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7f2905ffb640 1 -- 172.21.15.87:0/3535691046 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 453888674 0 0) 0x7f290002c630 con 0x7f29000c11b0 2024-04-26T02:21:34.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7f2905ffb640 1 -- 172.21.15.87:0/3535691046 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28f0003530 con 0x7f29000c11b0 2024-04-26T02:21:34.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7fb37dffb640 1 -- 172.21.15.87:0/1468073113 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 519466342 0 0) 0x7fb368003530 con 0x7fb3800af120 2024-04-26T02:21:34.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7f2905ffb640 1 -- 172.21.15.87:0/3535691046 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1529467376 0 0) 0x7f28f0003530 con 0x7f29000c11b0 2024-04-26T02:21:34.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7f2905ffb640 1 -- 172.21.15.87:0/3535691046 >> v1:172.21.15.87:6790/0 conn(0x7f290002a890 legacy=0x7f29000c0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7f2905ffb640 1 -- 172.21.15.87:0/3535691046 >> v1:172.21.15.122:6789/0 conn(0x7f2900034850 legacy=0x7f29000c6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7f2905ffb640 1 -- 172.21.15.87:0/3535691046 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f290002a090 con 0x7f29000c11b0 2024-04-26T02:21:34.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7f2905ffb640 1 -- 172.21.15.87:0/3535691046 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f28f8001620 con 0x7f29000c11b0 2024-04-26T02:21:34.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7fb37dffb640 1 -- 172.21.15.87:0/1468073113 >> v1:172.21.15.87:6790/0 conn(0x7fb3800af500 legacy=0x7fb38013e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7fb37dffb640 1 -- 172.21.15.87:0/1468073113 >> v1:172.21.15.122:6789/0 conn(0x7fb38013eb30 legacy=0x7fb380149810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7fb37dffb640 1 -- 172.21.15.87:0/1468073113 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb380070500 con 0x7fb3800af120 2024-04-26T02:21:34.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7fb6b401a2f0 con 0x7fb6bc034850 2024-04-26T02:21:34.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7fb37dffb640 1 -- 172.21.15.87:0/1468073113 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb370001620 con 0x7fb3800af120 2024-04-26T02:21:34.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7f2905ffb640 1 -- 172.21.15.87:0/3535691046 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f28f8001880 con 0x7f29000c11b0 2024-04-26T02:21:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7f2905ffb640 1 -- 172.21.15.87:0/3535691046 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f28f8001c10 con 0x7f29000c11b0 2024-04-26T02:21:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.477+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 4090+0+0 (unknown 2211391313 0 0) 0x7fb6b4080b70 con 0x7fb6bc034850 2024-04-26T02:21:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.481+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 <== mon.1 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7fb0dc0193f0 con 0x7fb0e4072250 2024-04-26T02:21:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.481+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 <== mon.1 v1:172.21.15.122:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 4090+0+0 (unknown 2211391313 0 0) 0x7fb0dc0800c0 con 0x7fb0e4072250 2024-04-26T02:21:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.481+0000 7f2907fff640 1 -- 172.21.15.87:0/3535691046 >> v1:172.21.15.87:6789/0 conn(0x7f29000c11b0 legacy=0x7f29000cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.481+0000 7fb37dffb640 1 -- 172.21.15.87:0/1468073113 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb3700018f0 con 0x7fb3800af120 2024-04-26T02:21:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.481+0000 7fb37ffff640 1 -- 172.21.15.87:0/1468073113 >> v1:172.21.15.87:6789/0 conn(0x7fb3800af120 legacy=0x7fb38013b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.481+0000 7fb37ffff640 1 -- 172.21.15.87:0/1468073113 shutdown_connections 2024-04-26T02:21:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.481+0000 7fb37ffff640 1 -- 172.21.15.87:0/1468073113 >> 172.21.15.87:0/1468073113 conn(0x7fb3800b0210 msgr2=0x7fb3800b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.481+0000 7fb37ffff640 1 -- 172.21.15.87:0/1468073113 shutdown_connections 2024-04-26T02:21:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f2907fff640 1 -- 172.21.15.87:0/3535691046 shutdown_connections 2024-04-26T02:21:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f2907fff640 1 -- 172.21.15.87:0/3535691046 >> 172.21.15.87:0/3535691046 conn(0x7f2900005900 msgr2=0x7f2900005d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f2907fff640 1 -- 172.21.15.87:0/3535691046 shutdown_connections 2024-04-26T02:21:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f2907fff640 1 -- 172.21.15.87:0/3535691046 wait complete. 2024-04-26T02:21:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.481+0000 7fb37ffff640 1 -- 172.21.15.87:0/1468073113 wait complete. 2024-04-26T02:21:34.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7fb37ffff640 1 Processor -- start 2024-04-26T02:21:34.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7fb37ffff640 1 -- start start 2024-04-26T02:21:34.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7fb37ffff640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3801513c0 con 0x7fb3800b25b0 2024-04-26T02:21:34.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7fb37ffff640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb38012c310 con 0x7fb38013eb30 2024-04-26T02:21:34.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7fb37ffff640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3801500a0 con 0x7fb3800b2bd0 2024-04-26T02:21:34.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7fb37e7fc640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fb3800b2bd0 0x7fb38012b4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40656/0 (socket says 172.21.15.87:40656) 2024-04-26T02:21:34.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7fb37e7fc640 1 -- 172.21.15.87:0/4242903528 learned_addr learned my addr 172.21.15.87:0/4242903528 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f2907fff640 1 Processor -- start 2024-04-26T02:21:34.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f2907fff640 1 -- start start 2024-04-26T02:21:34.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f2907fff640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2900000c50 con 0x7f290002a890 2024-04-26T02:21:34.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f2907fff640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29000b89c0 con 0x7f29000c11b0 2024-04-26T02:21:34.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f2907fff640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29000be120 con 0x7f2900034850 2024-04-26T02:21:34.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f29067fc640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f2900034850 0x7f29000b7ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40658/0 (socket says 172.21.15.87:40658) 2024-04-26T02:21:34.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f29067fc640 1 -- 172.21.15.87:0/1132059066 learned_addr learned my addr 172.21.15.87:0/1132059066 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1438862948 0 0) 0x7f2900000c50 con 0x7f290002a890 2024-04-26T02:21:34.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28ec003530 con 0x7f290002a890 2024-04-26T02:21:34.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 63228622 0 0) 0x7f28ec003530 con 0x7f290002a890 2024-04-26T02:21:34.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2900000c50 con 0x7f290002a890 2024-04-26T02:21:34.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.485+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f28fc0017f0 con 0x7f290002a890 2024-04-26T02:21:34.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.489+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 <== mon.1 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fb0dc0510e0 con 0x7fb0e4072250 2024-04-26T02:21:34.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.489+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4049381989 0 0) 0x7f29000b89c0 con 0x7f29000c11b0 2024-04-26T02:21:34.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.489+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28ec003530 con 0x7f29000c11b0 2024-04-26T02:21:34.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.489+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2848559861 0 0) 0x7f2900000c50 con 0x7f290002a890 2024-04-26T02:21:34.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.489+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 >> v1:172.21.15.87:6790/0 conn(0x7f2900034850 legacy=0x7f29000b7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.489+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 >> v1:172.21.15.122:6789/0 conn(0x7f29000c11b0 legacy=0x7f29000b82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.489+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29000c2130 con 0x7f290002a890 2024-04-26T02:21:34.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.489+0000 7f2907fff640 1 -- 172.21.15.87:0/1132059066 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f29000c1630 con 0x7f290002a890 2024-04-26T02:21:34.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.489+0000 7f2907fff640 1 -- 172.21.15.87:0/1132059066 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f29000c01c0 con 0x7f290002a890 2024-04-26T02:21:34.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.489+0000 7f2907fff640 1 -- 172.21.15.87:0/1132059066 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2900004ce0 con 0x7f290002a890 2024-04-26T02:21:34.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.489+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f28fc0036a0 con 0x7f290002a890 2024-04-26T02:21:34.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.489+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f28fc0039f0 con 0x7f290002a890 2024-04-26T02:21:34.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.489+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f28fc003b90 con 0x7f290002a890 2024-04-26T02:21:34.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.489+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 4090+0+0 (unknown 2211391313 0 0) 0x7f28fc07ff20 con 0x7f290002a890 2024-04-26T02:21:34.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.493+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2057510006 0 0) 0x7fb3801513c0 con 0x7fb3800b25b0 2024-04-26T02:21:34.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.493+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb36c003530 con 0x7fb3800b25b0 2024-04-26T02:21:34.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.493+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 177881384 0 0) 0x7fb38012c310 con 0x7fb38013eb30 2024-04-26T02:21:34.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.493+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3801513c0 con 0x7fb38013eb30 2024-04-26T02:21:34.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.493+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fb6bc0c7c80 con 0x7fb6bc034850 2024-04-26T02:21:34.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.497+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 818077148 0 0) 0x7fb3801500a0 con 0x7fb3800b2bd0 2024-04-26T02:21:34.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.497+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb38012c310 con 0x7fb3800b2bd0 2024-04-26T02:21:34.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.497+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2865118339 0 0) 0x7fb3801513c0 con 0x7fb38013eb30 2024-04-26T02:21:34.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.497+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3801500a0 con 0x7fb38013eb30 2024-04-26T02:21:34.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.497+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb370001900 con 0x7fb38013eb30 2024-04-26T02:21:34.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.497+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 <== mon.1 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2110521821 0 0) 0x7fb3801500a0 con 0x7fb38013eb30 2024-04-26T02:21:34.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.497+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 >> v1:172.21.15.87:6790/0 conn(0x7fb3800b2bd0 legacy=0x7fb38012b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.497+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 >> v1:172.21.15.87:6789/0 conn(0x7fb3800b25b0 legacy=0x7fb38012bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.497+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb380151630 con 0x7fb38013eb30 2024-04-26T02:21:34.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.497+0000 7fb37ffff640 1 -- 172.21.15.87:0/4242903528 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb3800b3060 con 0x7fb38013eb30 2024-04-26T02:21:34.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.497+0000 7fb37ffff640 1 -- 172.21.15.87:0/4242903528 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb380139440 con 0x7fb38013eb30 2024-04-26T02:21:34.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.497+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 <== mon.1 v1:172.21.15.122:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb370001b20 con 0x7fb38013eb30 2024-04-26T02:21:34.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.497+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 <== mon.1 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fb370003a50 con 0x7fb38013eb30 2024-04-26T02:21:34.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.497+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 <== mon.1 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7fb370003c70 con 0x7fb38013eb30 2024-04-26T02:21:34.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.501+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f28fc050f40 con 0x7f290002a890 2024-04-26T02:21:34.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.501+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 <== mon.1 v1:172.21.15.122:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 4090+0+0 (unknown 2211391313 0 0) 0x7fb37007fd40 con 0x7fb38013eb30 2024-04-26T02:21:34.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.509+0000 7fb37ffff640 1 -- 172.21.15.87:0/4242903528 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb360005090 con 0x7fb38013eb30 2024-04-26T02:21:34.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.521+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 <== mon.1 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fb370050d60 con 0x7fb38013eb30 2024-04-26T02:21:34.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.725+0000 7fe0aa0fd640 1 -- 172.21.15.87:0/3999536291 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fe084005380 con 0x7fe0a40af380 2024-04-26T02:21:34.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.725+0000 7fe08bfff640 1 -- 172.21.15.87:0/3999536291 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 3887400221) 0x7fe098051b20 con 0x7fe0a40af380 2024-04-26T02:21:34.727 INFO:teuthology.orchestra.run.smithi087.stdout:60129542148 2024-04-26T02:21:34.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.733+0000 7fe089ffb640 1 -- 172.21.15.87:0/3999536291 >> v1:172.21.15.122:6816/416193625 conn(0x7fe090006aa0 legacy=0x7fe090006f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.733+0000 7fe089ffb640 1 -- 172.21.15.87:0/3999536291 >> v1:172.21.15.87:6789/0 conn(0x7fe0a40af380 legacy=0x7fe0a41d5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.733+0000 7fe089ffb640 1 -- 172.21.15.87:0/3999536291 shutdown_connections 2024-04-26T02:21:34.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.733+0000 7fe089ffb640 1 -- 172.21.15.87:0/3999536291 >> 172.21.15.87:0/3999536291 conn(0x7fe0a40b0210 msgr2=0x7fe0a4074c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.733+0000 7fe089ffb640 1 -- 172.21.15.87:0/3999536291 shutdown_connections 2024-04-26T02:21:34.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.733+0000 7fe089ffb640 1 -- 172.21.15.87:0/3999536291 wait complete. 2024-04-26T02:21:34.752 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.3 2024-04-26T02:21:34.752 DEBUG:teuthology.parallel:result is None 2024-04-26T02:21:34.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.761+0000 7f78bfb8c640 1 -- 172.21.15.87:0/2499741650 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f78b8067bf0 con 0x7f78b813b4f0 2024-04-26T02:21:34.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.765+0000 7f78ae7fc640 1 -- 172.21.15.87:0/2499741650 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 13373408) 0x7f78b0051d20 con 0x7f78b813b4f0 2024-04-26T02:21:34.765 INFO:teuthology.orchestra.run.smithi087.stdout:60129542147 2024-04-26T02:21:34.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.769+0000 7f78bfb8c640 1 -- 172.21.15.87:0/2499741650 >> v1:172.21.15.122:6816/416193625 conn(0x7f789c068540 legacy=0x7f789c06aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.769+0000 7f78bfb8c640 1 -- 172.21.15.87:0/2499741650 >> v1:172.21.15.87:6790/0 conn(0x7f78b813b4f0 legacy=0x7f78b812d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.769+0000 7f78bfb8c640 1 -- 172.21.15.87:0/2499741650 shutdown_connections 2024-04-26T02:21:34.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.769+0000 7f78bfb8c640 1 -- 172.21.15.87:0/2499741650 >> 172.21.15.87:0/2499741650 conn(0x7f78b80b0210 msgr2=0x7f78b8141590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.769+0000 7f78bfb8c640 1 -- 172.21.15.87:0/2499741650 shutdown_connections 2024-04-26T02:21:34.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.773+0000 7f78bfb8c640 1 -- 172.21.15.87:0/2499741650 wait complete. 2024-04-26T02:21:34.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.773+0000 7f0d989aa640 1 -- 172.21.15.87:0/3745183772 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0d78005380 con 0x7f0d941586a0 2024-04-26T02:21:34.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.773+0000 7f0d82ffd640 1 -- 172.21.15.87:0/3745183772 <== mon.1 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 13373408) 0x7f0d8c051ad0 con 0x7f0d941586a0 2024-04-26T02:21:34.770 INFO:teuthology.orchestra.run.smithi087.stdout:60129542147 2024-04-26T02:21:34.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.773+0000 7f0d989aa640 1 -- 172.21.15.87:0/3745183772 >> v1:172.21.15.122:6816/416193625 conn(0x7f0d7c070ab0 legacy=0x7f0d7c072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.777+0000 7f0d989aa640 1 -- 172.21.15.87:0/3745183772 >> v1:172.21.15.122:6789/0 conn(0x7f0d941586a0 legacy=0x7f0d94089620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.777+0000 7f0d989aa640 1 -- 172.21.15.87:0/3745183772 shutdown_connections 2024-04-26T02:21:34.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.777+0000 7f0d989aa640 1 -- 172.21.15.87:0/3745183772 >> 172.21.15.87:0/3745183772 conn(0x7f0d940b0210 msgr2=0x7f0d940b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.777+0000 7f0d989aa640 1 -- 172.21.15.87:0/3745183772 shutdown_connections 2024-04-26T02:21:34.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.777+0000 7f0d989aa640 1 -- 172.21.15.87:0/3745183772 wait complete. 2024-04-26T02:21:34.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.785+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/309833182 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fbb3c00a8b0 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.785+0000 7fbb3affd640 1 -- 172.21.15.87:0/309833182 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 326014871) 0x7fbb40051910 con 0x7fbb3c0cbc10 2024-04-26T02:21:34.785 INFO:teuthology.orchestra.run.smithi087.stdout:60129542146 2024-04-26T02:21:34.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.785+0000 7fb6c3fff640 1 -- 172.21.15.87:0/2517792654 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fb6a4005380 con 0x7fb6bc034850 2024-04-26T02:21:34.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.789+0000 7fb6ab7fe640 1 -- 172.21.15.87:0/2517792654 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (unknown 1482059429 0 326014871) 0x7fb6b4051b90 con 0x7fb6bc034850 2024-04-26T02:21:34.786 INFO:teuthology.orchestra.run.smithi087.stdout:60129542146 2024-04-26T02:21:34.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.789+0000 7fb0ca7fc640 1 -- 172.21.15.87:0/997803590 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb0e41e0a50 con 0x7fb0e4072250 2024-04-26T02:21:34.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.789+0000 7fb0e0ff9640 1 -- 172.21.15.87:0/997803590 <== mon.1 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (unknown 1274345170 0 3056272221) 0x7fb0dc0509e0 con 0x7fb0e4072250 2024-04-26T02:21:34.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.789+0000 7fb6c3fff640 1 -- 172.21.15.87:0/2517792654 >> v1:172.21.15.122:6816/416193625 conn(0x7fb6b0068680 legacy=0x7fb6b006ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.788 INFO:teuthology.orchestra.run.smithi087.stdout:55834574852 2024-04-26T02:21:34.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.789+0000 7fb6c3fff640 1 -- 172.21.15.87:0/2517792654 >> v1:172.21.15.87:6790/0 conn(0x7fb6bc034850 legacy=0x7fb6bc1697c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.789+0000 7fb6c3fff640 1 -- 172.21.15.87:0/2517792654 shutdown_connections 2024-04-26T02:21:34.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.789+0000 7fb6c3fff640 1 -- 172.21.15.87:0/2517792654 >> 172.21.15.87:0/2517792654 conn(0x7fb6bc005900 msgr2=0x7fb6bc0c6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.789+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/309833182 >> v1:172.21.15.122:6816/416193625 conn(0x7fbb28068310 legacy=0x7fbb2806a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.789+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/309833182 >> v1:172.21.15.87:6789/0 conn(0x7fbb3c0cbc10 legacy=0x7fbb3c0bd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.793+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/309833182 shutdown_connections 2024-04-26T02:21:34.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.793+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/309833182 >> 172.21.15.87:0/309833182 conn(0x7fbb3c005c70 msgr2=0x7fbb3c004830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.793+0000 7fb6c3fff640 1 -- 172.21.15.87:0/2517792654 shutdown_connections 2024-04-26T02:21:34.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.793+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/309833182 shutdown_connections 2024-04-26T02:21:34.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.793+0000 7fb6c3fff640 1 -- 172.21.15.87:0/2517792654 wait complete. 2024-04-26T02:21:34.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.793+0000 7fbb4b0b7640 1 -- 172.21.15.87:0/309833182 wait complete. 2024-04-26T02:21:34.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.793+0000 7fb0ca7fc640 1 -- 172.21.15.87:0/997803590 >> v1:172.21.15.122:6816/416193625 conn(0x7fb0d00684c0 legacy=0x7fb0d006a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.793+0000 7fb0ca7fc640 1 -- 172.21.15.87:0/997803590 >> v1:172.21.15.122:6789/0 conn(0x7fb0e4072250 legacy=0x7fb0e41e7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.793+0000 7fb0ca7fc640 1 -- 172.21.15.87:0/997803590 shutdown_connections 2024-04-26T02:21:34.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.793+0000 7fb0ca7fc640 1 -- 172.21.15.87:0/997803590 >> 172.21.15.87:0/997803590 conn(0x7fb0e40b4230 msgr2=0x7fb0e40acd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.793+0000 7fb0ca7fc640 1 -- 172.21.15.87:0/997803590 shutdown_connections 2024-04-26T02:21:34.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.793+0000 7fb0ca7fc640 1 -- 172.21.15.87:0/997803590 wait complete. 2024-04-26T02:21:34.797 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 60129542147 for osd.5 2024-04-26T02:21:34.797 DEBUG:teuthology.parallel:result is None 2024-04-26T02:21:34.798 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 60129542147 for osd.0 2024-04-26T02:21:34.798 DEBUG:teuthology.parallel:result is None 2024-04-26T02:21:34.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.813+0000 7f2907fff640 1 -- 172.21.15.87:0/1132059066 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f29000c04f0 con 0x7f290002a890 2024-04-26T02:21:34.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.813+0000 7f28f77fe640 1 -- 172.21.15.87:0/1132059066 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 13373408) 0x7f28fc050840 con 0x7f290002a890 2024-04-26T02:21:34.811 INFO:teuthology.orchestra.run.smithi087.stdout:60129542147 2024-04-26T02:21:34.813 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542146 got 60129542146 for osd.1 2024-04-26T02:21:34.813 DEBUG:teuthology.parallel:result is None 2024-04-26T02:21:34.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.813+0000 7f2907fff640 1 -- 172.21.15.87:0/1132059066 >> v1:172.21.15.122:6816/416193625 conn(0x7f28ec0684d0 legacy=0x7f28ec06a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.813+0000 7f2907fff640 1 -- 172.21.15.87:0/1132059066 >> v1:172.21.15.87:6789/0 conn(0x7f290002a890 legacy=0x7f290015d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.813+0000 7f2907fff640 1 -- 172.21.15.87:0/1132059066 shutdown_connections 2024-04-26T02:21:34.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.813+0000 7f2907fff640 1 -- 172.21.15.87:0/1132059066 >> 172.21.15.87:0/1132059066 conn(0x7f2900005900 msgr2=0x7f29000373d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.817+0000 7f2907fff640 1 -- 172.21.15.87:0/1132059066 shutdown_connections 2024-04-26T02:21:34.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.817+0000 7f2907fff640 1 -- 172.21.15.87:0/1132059066 wait complete. 2024-04-26T02:21:34.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.817+0000 7fb37ffff640 1 -- 172.21.15.87:0/4242903528 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fb360005380 con 0x7fb38013eb30 2024-04-26T02:21:34.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.817+0000 7fb3677fe640 1 -- 172.21.15.87:0/4242903528 <== mon.1 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 3056272221) 0x7fb370050660 con 0x7fb38013eb30 2024-04-26T02:21:34.815 INFO:teuthology.orchestra.run.smithi087.stdout:55834574852 2024-04-26T02:21:34.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.817+0000 7fb37ffff640 1 -- 172.21.15.87:0/4242903528 >> v1:172.21.15.122:6816/416193625 conn(0x7fb36c068530 legacy=0x7fb36c06a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.817+0000 7fb37ffff640 1 -- 172.21.15.87:0/4242903528 >> v1:172.21.15.122:6789/0 conn(0x7fb38013eb30 legacy=0x7fb38014f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.817+0000 7fb37ffff640 1 -- 172.21.15.87:0/4242903528 shutdown_connections 2024-04-26T02:21:34.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.817+0000 7fb37ffff640 1 -- 172.21.15.87:0/4242903528 >> 172.21.15.87:0/4242903528 conn(0x7fb3800b0210 msgr2=0x7fb380130180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.817+0000 7fb37ffff640 1 -- 172.21.15.87:0/4242903528 shutdown_connections 2024-04-26T02:21:34.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.817+0000 7fb37ffff640 1 -- 172.21.15.87:0/4242903528 wait complete. 2024-04-26T02:21:34.818 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.6 2024-04-26T02:21:34.818 DEBUG:teuthology.parallel:result is None 2024-04-26T02:21:34.819 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542146 got 60129542146 for osd.7 2024-04-26T02:21:34.819 DEBUG:teuthology.parallel:result is None 2024-04-26T02:21:34.831 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 60129542147 for osd.4 2024-04-26T02:21:34.831 DEBUG:teuthology.parallel:result is None 2024-04-26T02:21:34.833 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.2 2024-04-26T02:21:34.833 DEBUG:teuthology.parallel:result is None 2024-04-26T02:21:34.833 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T02:21:34.833 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:21:34.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.973+0000 7f535e550640 1 Processor -- start 2024-04-26T02:21:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.973+0000 7f535e550640 1 -- start start 2024-04-26T02:21:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.973+0000 7f535e550640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53580ab1a0 con 0x7f535813f9e0 2024-04-26T02:21:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.973+0000 7f535e550640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53580a9a10 con 0x7f53580a9e60 2024-04-26T02:21:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.973+0000 7f535e550640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53580a8fd0 con 0x7f53580a8bf0 2024-04-26T02:21:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f535caed640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f535813f9e0 0x7f535814d940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:54040/0 (socket says 172.21.15.87:54040) 2024-04-26T02:21:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f535caed640 1 -- 172.21.15.87:0/2331122643 learned_addr learned my addr 172.21.15.87:0/2331122643 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5357fff640 1 --1- 172.21.15.87:0/2331122643 >> v1:172.21.15.87:6790/0 conn(0x7f53580a8bf0 0x7f535813f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40674/0 (socket says 172.21.15.87:40674) 2024-04-26T02:21:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5356ffd640 1 -- 172.21.15.87:0/2331122643 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3001653285 0 0) 0x7f53580a8fd0 con 0x7f53580a8bf0 2024-04-26T02:21:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5356ffd640 1 -- 172.21.15.87:0/2331122643 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5340003530 con 0x7f53580a8bf0 2024-04-26T02:21:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5356ffd640 1 -- 172.21.15.87:0/2331122643 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1741863273 0 0) 0x7f53580ab1a0 con 0x7f535813f9e0 2024-04-26T02:21:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5356ffd640 1 -- 172.21.15.87:0/2331122643 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53580a8fd0 con 0x7f535813f9e0 2024-04-26T02:21:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5356ffd640 1 -- 172.21.15.87:0/2331122643 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3029779123 0 0) 0x7f5340003530 con 0x7f53580a8bf0 2024-04-26T02:21:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5356ffd640 1 -- 172.21.15.87:0/2331122643 >> v1:172.21.15.122:6789/0 conn(0x7f53580a9e60 legacy=0x7f5358145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5356ffd640 1 -- 172.21.15.87:0/2331122643 >> v1:172.21.15.87:6789/0 conn(0x7f535813f9e0 legacy=0x7f535814d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5356ffd640 1 -- 172.21.15.87:0/2331122643 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53580a91a0 con 0x7f53580a8bf0 2024-04-26T02:21:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5356ffd640 1 -- 172.21.15.87:0/2331122643 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f53500018d0 con 0x7f53580a8bf0 2024-04-26T02:21:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5356ffd640 1 -- 172.21.15.87:0/2331122643 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5350001b40 con 0x7f53580a8bf0 2024-04-26T02:21:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5356ffd640 1 -- 172.21.15.87:0/2331122643 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f5350001f10 con 0x7f53580a8bf0 2024-04-26T02:21:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f535e550640 1 -- 172.21.15.87:0/2331122643 >> v1:172.21.15.87:6790/0 conn(0x7f53580a8bf0 legacy=0x7f535813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f535e550640 1 -- 172.21.15.87:0/2331122643 shutdown_connections 2024-04-26T02:21:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f535e550640 1 -- 172.21.15.87:0/2331122643 >> 172.21.15.87:0/2331122643 conn(0x7f53580b7af0 msgr2=0x7f53580b7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f535e550640 1 -- 172.21.15.87:0/2331122643 shutdown_connections 2024-04-26T02:21:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f535e550640 1 -- 172.21.15.87:0/2331122643 wait complete. 2024-04-26T02:21:34.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f535e550640 1 Processor -- start 2024-04-26T02:21:34.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f535e550640 1 -- start start 2024-04-26T02:21:34.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f535e550640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53580ad7f0 con 0x7f535813f9e0 2024-04-26T02:21:34.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f535e550640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53580ada60 con 0x7f53581384b0 2024-04-26T02:21:34.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f535e550640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53581db740 con 0x7f53580a8bf0 2024-04-26T02:21:34.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5357fff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f53580a8bf0 0x7f53581dac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40686/0 (socket says 172.21.15.87:40686) 2024-04-26T02:21:34.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5357fff640 1 -- 172.21.15.87:0/1826850893 learned_addr learned my addr 172.21.15.87:0/1826850893 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:34.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1774924131 0 0) 0x7f53581db740 con 0x7f53580a8bf0 2024-04-26T02:21:34.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5344003530 con 0x7f53580a8bf0 2024-04-26T02:21:34.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4126651259 0 0) 0x7f53580ad7f0 con 0x7f535813f9e0 2024-04-26T02:21:34.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53581db740 con 0x7f535813f9e0 2024-04-26T02:21:34.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 768889981 0 0) 0x7f5344003530 con 0x7f53580a8bf0 2024-04-26T02:21:34.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53580ad7f0 con 0x7f53580a8bf0 2024-04-26T02:21:34.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f5350001f10 con 0x7f53580a8bf0 2024-04-26T02:21:34.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 664961806 0 0) 0x7f53581db740 con 0x7f535813f9e0 2024-04-26T02:21:34.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5344003530 con 0x7f535813f9e0 2024-04-26T02:21:34.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f534c001840 con 0x7f535813f9e0 2024-04-26T02:21:34.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4215834439 0 0) 0x7f53580ad7f0 con 0x7f53580a8bf0 2024-04-26T02:21:34.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 >> v1:172.21.15.122:6789/0 conn(0x7f53581384b0 legacy=0x7f53580ab830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 >> v1:172.21.15.87:6789/0 conn(0x7f535813f9e0 legacy=0x7f5358137da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:34.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5358142dd0 con 0x7f53580a8bf0 2024-04-26T02:21:34.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f535e550640 1 -- 172.21.15.87:0/1826850893 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f53581db330 con 0x7f53580a8bf0 2024-04-26T02:21:34.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f535e550640 1 -- 172.21.15.87:0/1826850893 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f53580ac110 con 0x7f53580a8bf0 2024-04-26T02:21:34.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5350001b40 con 0x7f53580a8bf0 2024-04-26T02:21:34.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.977+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f5350004fc0 con 0x7f53580a8bf0 2024-04-26T02:21:34.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.981+0000 7f535e550640 1 -- 172.21.15.87:0/1826850893 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5338005090 con 0x7f53580a8bf0 2024-04-26T02:21:34.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.981+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f5350005160 con 0x7f53580a8bf0 2024-04-26T02:21:34.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.981+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 4090+0+0 (unknown 2211391313 0 0) 0x7f5350081530 con 0x7f53580a8bf0 2024-04-26T02:21:34.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:34.985+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f5350052550 con 0x7f53580a8bf0 2024-04-26T02:21:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.129+0000 7f535e550640 1 -- 172.21.15.87:0/1826850893 --> v1:172.21.15.122:6816/416193625 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5338000ba0 con 0x7f53440684d0 2024-04-26T02:21:35.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.133+0000 7f5354ff9640 1 -- 172.21.15.87:0/1826850893 <== mgr.4108 v1:172.21.15.122:6816/416193625 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35513 (unknown 2965378022 0 1923566276) 0x7f5338000ba0 con 0x7f53440684d0 2024-04-26T02:21:35.133 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:21:35.133 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-26T02:21:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.137+0000 7f535e550640 1 -- 172.21.15.87:0/1826850893 >> v1:172.21.15.122:6816/416193625 conn(0x7f53440684d0 legacy=0x7f534406a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.137+0000 7f535e550640 1 -- 172.21.15.87:0/1826850893 >> v1:172.21.15.87:6790/0 conn(0x7f53580a8bf0 legacy=0x7f53581dac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.137+0000 7f535e550640 1 -- 172.21.15.87:0/1826850893 shutdown_connections 2024-04-26T02:21:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.137+0000 7f535e550640 1 -- 172.21.15.87:0/1826850893 >> 172.21.15.87:0/1826850893 conn(0x7f53580b7af0 msgr2=0x7f5358074c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.137+0000 7f535e550640 1 -- 172.21.15.87:0/1826850893 shutdown_connections 2024-04-26T02:21:35.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.137+0000 7f535e550640 1 -- 172.21.15.87:0/1826850893 wait complete. 2024-04-26T02:21:35.149 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-04-26T02:21:33.441225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":217448,"kb_used_data":1280,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757272,"statfs":{"total":773094113280,"available":772871446528,"internally_reserved":0,"allocated":1310720,"data_stored":1432294,"data_compressed":11166,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.104668"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-26T02:21:32.418783+0000","last_change":"2024-04-26T02:21:31.429340+0000","last_active":"2024-04-26T02:21:32.418783+0000","last_peered":"2024-04-26T02:21:32.418783+0000","last_clean":"2024-04-26T02:21:32.418783+0000","last_became_active":"2024-04-26T02:21:31.428657+0000","last_became_peered":"2024-04-26T02:21:31.428657+0000","last_unstale":"2024-04-26T02:21:32.418783+0000","last_undegraded":"2024-04-26T02:21:32.418783+0000","last_fullsized":"2024-04-26T02:21:32.418783+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:53:35.800975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-26T02:21:32.423394+0000","last_change":"2024-04-26T02:21:31.432330+0000","last_active":"2024-04-26T02:21:32.423394+0000","last_peered":"2024-04-26T02:21:32.423394+0000","last_clean":"2024-04-26T02:21:32.423394+0000","last_became_active":"2024-04-26T02:21:31.431958+0000","last_became_peered":"2024-04-26T02:21:31.431958+0000","last_unstale":"2024-04-26T02:21:32.423394+0000","last_undegraded":"2024-04-26T02:21:32.423394+0000","last_fullsized":"2024-04-26T02:21:32.423394+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:36:20.588651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-26T02:21:31.433924+0000","last_change":"2024-04-26T02:21:31.430988+0000","last_active":"2024-04-26T02:21:31.433924+0000","last_peered":"2024-04-26T02:21:31.433924+0000","last_clean":"2024-04-26T02:21:31.433924+0000","last_became_active":"2024-04-26T02:21:31.430227+0000","last_became_peered":"2024-04-26T02:21:31.430227+0000","last_unstale":"2024-04-26T02:21:31.433924+0000","last_undegraded":"2024-04-26T02:21:31.433924+0000","last_fullsized":"2024-04-26T02:21:31.433924+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:26:15.568270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-26T02:21:32.423358+0000","last_change":"2024-04-26T02:21:31.431358+0000","last_active":"2024-04-26T02:21:32.423358+0000","last_peered":"2024-04-26T02:21:32.423358+0000","last_clean":"2024-04-26T02:21:32.423358+0000","last_became_active":"2024-04-26T02:21:31.431071+0000","last_became_peered":"2024-04-26T02:21:31.431071+0000","last_unstale":"2024-04-26T02:21:32.423358+0000","last_undegraded":"2024-04-26T02:21:32.423358+0000","last_fullsized":"2024-04-26T02:21:32.423358+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:18:58.049430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-26T02:21:32.429207+0000","last_change":"2024-04-26T02:21:32.425721+0000","last_active":"2024-04-26T02:21:32.429207+0000","last_peered":"2024-04-26T02:21:32.429207+0000","last_clean":"2024-04-26T02:21:32.429207+0000","last_became_active":"2024-04-26T02:21:32.425189+0000","last_became_peered":"2024-04-26T02:21:32.425189+0000","last_unstale":"2024-04-26T02:21:32.429207+0000","last_undegraded":"2024-04-26T02:21:32.429207+0000","last_fullsized":"2024-04-26T02:21:32.429207+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:07:31.466761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-26T02:21:33.340196+0000","last_change":"2024-04-26T02:21:31.436585+0000","last_active":"2024-04-26T02:21:33.340196+0000","last_peered":"2024-04-26T02:21:33.340196+0000","last_clean":"2024-04-26T02:21:33.340196+0000","last_became_active":"2024-04-26T02:21:31.436226+0000","last_became_peered":"2024-04-26T02:21:31.436226+0000","last_unstale":"2024-04-26T02:21:33.340196+0000","last_undegraded":"2024-04-26T02:21:33.340196+0000","last_fullsized":"2024-04-26T02:21:33.340196+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:05.168797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-26T02:21:31.434385+0000","last_change":"2024-04-26T02:21:31.433313+0000","last_active":"2024-04-26T02:21:31.434385+0000","last_peered":"2024-04-26T02:21:31.434385+0000","last_clean":"2024-04-26T02:21:31.434385+0000","last_became_active":"2024-04-26T02:21:31.432395+0000","last_became_peered":"2024-04-26T02:21:31.432395+0000","last_unstale":"2024-04-26T02:21:31.434385+0000","last_undegraded":"2024-04-26T02:21:31.434385+0000","last_fullsized":"2024-04-26T02:21:31.434385+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:12:38.727950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":4,"reported_epoch":18,"state":"creating+activating","last_fresh":"2024-04-26T02:21:32.423139+0000","last_change":"2024-04-26T02:21:32.423139+0000","last_active":"2024-04-26T02:21:30.392677+0000","last_peered":"2024-04-26T02:21:30.392677+0000","last_clean":"2024-04-26T02:21:30.392677+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T02:21:32.423139+0000","last_undegraded":"2024-04-26T02:21:32.423139+0000","last_fullsized":"2024-04-26T02:21:32.423139+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-26T02:21:31.424214+0000","last_change":"2024-04-26T02:21:29.434185+0000","last_active":"2024-04-26T02:21:31.424214+0000","last_peered":"2024-04-26T02:21:31.424214+0000","last_clean":"2024-04-26T02:21:31.424214+0000","last_became_active":"2024-04-26T02:21:29.432363+0000","last_became_peered":"2024-04-26T02:21:29.432363+0000","last_unstale":"2024-04-26T02:21:31.424214+0000","last_undegraded":"2024-04-26T02:21:31.424214+0000","last_fullsized":"2024-04-26T02:21:31.424214+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:28.375870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:28.375870+0000","last_clean_scrub_stamp":"2024-04-26T02:21:28.375870+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:18:11.058445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11166,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542146,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":574121,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":49491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":45193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":147456,"data_stored":45193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542146,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":49491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542147,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":200704,"data_stored":574121,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":49491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":45193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:21:35.150 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:21:35.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.269+0000 7f269e532640 1 Processor -- start 2024-04-26T02:21:35.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.269+0000 7f269e532640 1 -- start start 2024-04-26T02:21:35.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.269+0000 7f269e532640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26980a9f60 con 0x7f269813faf0 2024-04-26T02:21:35.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.269+0000 7f269e532640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26980aadd0 con 0x7f26980b23f0 2024-04-26T02:21:35.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.269+0000 7f269e532640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26980ab0b0 con 0x7f26980a8b70 2024-04-26T02:21:35.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2697fff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f26980a8b70 0x7f26981427e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40688/0 (socket says 172.21.15.87:40688) 2024-04-26T02:21:35.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f269cacf640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f269813faf0 0x7f269814da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:54066/0 (socket says 172.21.15.87:54066) 2024-04-26T02:21:35.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2697fff640 1 -- 172.21.15.87:0/3343700073 learned_addr learned my addr 172.21.15.87:0/3343700073 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:35.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2696ffd640 1 -- 172.21.15.87:0/3343700073 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3642209015 0 0) 0x7f26980ab0b0 con 0x7f26980a8b70 2024-04-26T02:21:35.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2696ffd640 1 -- 172.21.15.87:0/3343700073 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2680003530 con 0x7f26980a8b70 2024-04-26T02:21:35.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2696ffd640 1 -- 172.21.15.87:0/3343700073 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1058267599 0 0) 0x7f26980a9f60 con 0x7f269813faf0 2024-04-26T02:21:35.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2696ffd640 1 -- 172.21.15.87:0/3343700073 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26980ab0b0 con 0x7f269813faf0 2024-04-26T02:21:35.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2696ffd640 1 -- 172.21.15.87:0/3343700073 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3516954037 0 0) 0x7f26980aadd0 con 0x7f26980b23f0 2024-04-26T02:21:35.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2696ffd640 1 -- 172.21.15.87:0/3343700073 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26980a9f60 con 0x7f26980b23f0 2024-04-26T02:21:35.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2696ffd640 1 -- 172.21.15.87:0/3343700073 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 108691009 0 0) 0x7f2680003530 con 0x7f26980a8b70 2024-04-26T02:21:35.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2696ffd640 1 -- 172.21.15.87:0/3343700073 >> v1:172.21.15.122:6789/0 conn(0x7f26980b23f0 legacy=0x7f269813f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2696ffd640 1 -- 172.21.15.87:0/3343700073 >> v1:172.21.15.87:6789/0 conn(0x7f269813faf0 legacy=0x7f269814da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2696ffd640 1 -- 172.21.15.87:0/3343700073 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26981422c0 con 0x7f26980a8b70 2024-04-26T02:21:35.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2696ffd640 1 -- 172.21.15.87:0/3343700073 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f268c001620 con 0x7f26980a8b70 2024-04-26T02:21:35.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2696ffd640 1 -- 172.21.15.87:0/3343700073 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f268c001880 con 0x7f26980a8b70 2024-04-26T02:21:35.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2696ffd640 1 -- 172.21.15.87:0/3343700073 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f268c001c10 con 0x7f26980a8b70 2024-04-26T02:21:35.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f269e532640 1 -- 172.21.15.87:0/3343700073 >> v1:172.21.15.87:6790/0 conn(0x7f26980a8b70 legacy=0x7f26981427e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f269e532640 1 -- 172.21.15.87:0/3343700073 shutdown_connections 2024-04-26T02:21:35.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f269e532640 1 -- 172.21.15.87:0/3343700073 >> 172.21.15.87:0/3343700073 conn(0x7f26980b7a70 msgr2=0x7f26980b7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:35.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f269e532640 1 -- 172.21.15.87:0/3343700073 shutdown_connections 2024-04-26T02:21:35.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f269e532640 1 -- 172.21.15.87:0/3343700073 wait complete. 2024-04-26T02:21:35.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f269e532640 1 Processor -- start 2024-04-26T02:21:35.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f269e532640 1 -- start start 2024-04-26T02:21:35.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f269e532640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26980ac720 con 0x7f26980a8b70 2024-04-26T02:21:35.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f269e532640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26980b1650 con 0x7f269813faf0 2024-04-26T02:21:35.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f269e532640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26980b1820 con 0x7f26980b23f0 2024-04-26T02:21:35.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f26977fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f26980b23f0 0x7f26981d7310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40698/0 (socket says 172.21.15.87:40698) 2024-04-26T02:21:35.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f26977fe640 1 -- 172.21.15.87:0/916262351 learned_addr learned my addr 172.21.15.87:0/916262351 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:35.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 832555963 0 0) 0x7f26980b1820 con 0x7f26980b23f0 2024-04-26T02:21:35.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2684003530 con 0x7f26980b23f0 2024-04-26T02:21:35.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 110045106 0 0) 0x7f26980ac720 con 0x7f26980a8b70 2024-04-26T02:21:35.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26980b1820 con 0x7f26980a8b70 2024-04-26T02:21:35.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 666554683 0 0) 0x7f2684003530 con 0x7f26980b23f0 2024-04-26T02:21:35.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26980ac720 con 0x7f26980b23f0 2024-04-26T02:21:35.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f26880019a0 con 0x7f26980b23f0 2024-04-26T02:21:35.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1938522703 0 0) 0x7f26980b1820 con 0x7f26980a8b70 2024-04-26T02:21:35.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2684003530 con 0x7f26980a8b70 2024-04-26T02:21:35.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f268c001380 con 0x7f26980a8b70 2024-04-26T02:21:35.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 736674808 0 0) 0x7f26980ac720 con 0x7f26980b23f0 2024-04-26T02:21:35.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 >> v1:172.21.15.122:6789/0 conn(0x7f269813faf0 legacy=0x7f26981d7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.273+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 >> v1:172.21.15.87:6789/0 conn(0x7f26980a8b70 legacy=0x7f26980b0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.277+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26981451c0 con 0x7f26980b23f0 2024-04-26T02:21:35.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.277+0000 7f269e532640 1 -- 172.21.15.87:0/916262351 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f26981453f0 con 0x7f26980b23f0 2024-04-26T02:21:35.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.277+0000 7f269e532640 1 -- 172.21.15.87:0/916262351 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2698146340 con 0x7f26980b23f0 2024-04-26T02:21:35.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.277+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2688001cc0 con 0x7f26980b23f0 2024-04-26T02:21:35.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.277+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f2688003a70 con 0x7f26980b23f0 2024-04-26T02:21:35.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.277+0000 7f269e532640 1 -- 172.21.15.87:0/916262351 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2678005090 con 0x7f26980b23f0 2024-04-26T02:21:35.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.277+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f2688003c90 con 0x7f26980b23f0 2024-04-26T02:21:35.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.277+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 4090+0+0 (unknown 2211391313 0 0) 0x7f2688081120 con 0x7f26980b23f0 2024-04-26T02:21:35.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.281+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f26880521a0 con 0x7f26980b23f0 2024-04-26T02:21:35.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.425+0000 7f269e532640 1 -- 172.21.15.87:0/916262351 --> v1:172.21.15.122:6816/416193625 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2678000ba0 con 0x7f26840682b0 2024-04-26T02:21:35.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.429+0000 7f2694ff9640 1 -- 172.21.15.87:0/916262351 <== mgr.4108 v1:172.21.15.122:6816/416193625 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35588 (unknown 2965378022 0 2939930680) 0x7f2678000ba0 con 0x7f26840682b0 2024-04-26T02:21:35.427 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:21:35.428 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-26T02:21:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.429+0000 7f269e532640 1 -- 172.21.15.87:0/916262351 >> v1:172.21.15.122:6816/416193625 conn(0x7f26840682b0 legacy=0x7f268406a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.429+0000 7f269e532640 1 -- 172.21.15.87:0/916262351 >> v1:172.21.15.87:6790/0 conn(0x7f26980b23f0 legacy=0x7f26981d7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.429+0000 7f269e532640 1 -- 172.21.15.87:0/916262351 shutdown_connections 2024-04-26T02:21:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.429+0000 7f269e532640 1 -- 172.21.15.87:0/916262351 >> 172.21.15.87:0/916262351 conn(0x7f26980b7a70 msgr2=0x7f2698074c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.433+0000 7f269e532640 1 -- 172.21.15.87:0/916262351 shutdown_connections 2024-04-26T02:21:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.433+0000 7f269e532640 1 -- 172.21.15.87:0/916262351 wait complete. 2024-04-26T02:21:35.444 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-26T02:21:35.337190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":217484,"kb_used_data":1304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757236,"statfs":{"total":773094113280,"available":772871409664,"internally_reserved":0,"allocated":1335296,"data_stored":1445518,"data_compressed":11166,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000634"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-26T02:21:32.418783+0000","last_change":"2024-04-26T02:21:31.429340+0000","last_active":"2024-04-26T02:21:32.418783+0000","last_peered":"2024-04-26T02:21:32.418783+0000","last_clean":"2024-04-26T02:21:32.418783+0000","last_became_active":"2024-04-26T02:21:31.428657+0000","last_became_peered":"2024-04-26T02:21:31.428657+0000","last_unstale":"2024-04-26T02:21:32.418783+0000","last_undegraded":"2024-04-26T02:21:32.418783+0000","last_fullsized":"2024-04-26T02:21:32.418783+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:53:35.800975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-26T02:21:32.423394+0000","last_change":"2024-04-26T02:21:31.432330+0000","last_active":"2024-04-26T02:21:32.423394+0000","last_peered":"2024-04-26T02:21:32.423394+0000","last_clean":"2024-04-26T02:21:32.423394+0000","last_became_active":"2024-04-26T02:21:31.431958+0000","last_became_peered":"2024-04-26T02:21:31.431958+0000","last_unstale":"2024-04-26T02:21:32.423394+0000","last_undegraded":"2024-04-26T02:21:32.423394+0000","last_fullsized":"2024-04-26T02:21:32.423394+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:36:20.588651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-26T02:21:33.439573+0000","last_change":"2024-04-26T02:21:31.430988+0000","last_active":"2024-04-26T02:21:33.439573+0000","last_peered":"2024-04-26T02:21:33.439573+0000","last_clean":"2024-04-26T02:21:33.439573+0000","last_became_active":"2024-04-26T02:21:31.430227+0000","last_became_peered":"2024-04-26T02:21:31.430227+0000","last_unstale":"2024-04-26T02:21:33.439573+0000","last_undegraded":"2024-04-26T02:21:33.439573+0000","last_fullsized":"2024-04-26T02:21:33.439573+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:32:21.265195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-26T02:21:32.423358+0000","last_change":"2024-04-26T02:21:31.431358+0000","last_active":"2024-04-26T02:21:32.423358+0000","last_peered":"2024-04-26T02:21:32.423358+0000","last_clean":"2024-04-26T02:21:32.423358+0000","last_became_active":"2024-04-26T02:21:31.431071+0000","last_became_peered":"2024-04-26T02:21:31.431071+0000","last_unstale":"2024-04-26T02:21:32.423358+0000","last_undegraded":"2024-04-26T02:21:32.423358+0000","last_fullsized":"2024-04-26T02:21:32.423358+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:18:58.049430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-26T02:21:34.214491+0000","last_change":"2024-04-26T02:21:32.425721+0000","last_active":"2024-04-26T02:21:34.214491+0000","last_peered":"2024-04-26T02:21:34.214491+0000","last_clean":"2024-04-26T02:21:34.214491+0000","last_became_active":"2024-04-26T02:21:32.425189+0000","last_became_peered":"2024-04-26T02:21:32.425189+0000","last_unstale":"2024-04-26T02:21:34.214491+0000","last_undegraded":"2024-04-26T02:21:34.214491+0000","last_fullsized":"2024-04-26T02:21:34.214491+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:56:47.156837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-26T02:21:33.340196+0000","last_change":"2024-04-26T02:21:31.436585+0000","last_active":"2024-04-26T02:21:33.340196+0000","last_peered":"2024-04-26T02:21:33.340196+0000","last_clean":"2024-04-26T02:21:33.340196+0000","last_became_active":"2024-04-26T02:21:31.436226+0000","last_became_peered":"2024-04-26T02:21:31.436226+0000","last_unstale":"2024-04-26T02:21:33.340196+0000","last_undegraded":"2024-04-26T02:21:33.340196+0000","last_fullsized":"2024-04-26T02:21:33.340196+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:05.168797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-26T02:21:33.439565+0000","last_change":"2024-04-26T02:21:31.433313+0000","last_active":"2024-04-26T02:21:33.439565+0000","last_peered":"2024-04-26T02:21:33.439565+0000","last_clean":"2024-04-26T02:21:33.439565+0000","last_became_active":"2024-04-26T02:21:31.432395+0000","last_became_peered":"2024-04-26T02:21:31.432395+0000","last_unstale":"2024-04-26T02:21:33.439565+0000","last_undegraded":"2024-04-26T02:21:33.439565+0000","last_fullsized":"2024-04-26T02:21:33.439565+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:57:44.139176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-26T02:21:34.214478+0000","last_change":"2024-04-26T02:21:33.341715+0000","last_active":"2024-04-26T02:21:34.214478+0000","last_peered":"2024-04-26T02:21:34.214478+0000","last_clean":"2024-04-26T02:21:34.214478+0000","last_became_active":"2024-04-26T02:21:33.340937+0000","last_became_peered":"2024-04-26T02:21:33.340937+0000","last_unstale":"2024-04-26T02:21:34.214478+0000","last_undegraded":"2024-04-26T02:21:34.214478+0000","last_fullsized":"2024-04-26T02:21:34.214478+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:03:17.530982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-26T02:21:33.439465+0000","last_change":"2024-04-26T02:21:29.434185+0000","last_active":"2024-04-26T02:21:33.439465+0000","last_peered":"2024-04-26T02:21:33.439465+0000","last_clean":"2024-04-26T02:21:33.439465+0000","last_became_active":"2024-04-26T02:21:29.432363+0000","last_became_peered":"2024-04-26T02:21:29.432363+0000","last_unstale":"2024-04-26T02:21:33.439465+0000","last_undegraded":"2024-04-26T02:21:33.439465+0000","last_fullsized":"2024-04-26T02:21:33.439465+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:28.375870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:28.375870+0000","last_clean_scrub_stamp":"2024-04-26T02:21:28.375870+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:18:11.058445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11166,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542147,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":574121,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":49491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":49491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":147456,"data_stored":45193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542147,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":49491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":583047,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":49491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":45193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:21:35.445 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T02:21:35.445 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T02:21:35.445 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-26T02:21:35.445 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-26T02:21:35.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c0c78e640 1 Processor -- start 2024-04-26T02:21:35.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c0c78e640 1 -- start start 2024-04-26T02:21:35.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c0c78e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c040a8990 con 0x7f8c040ae8e0 2024-04-26T02:21:35.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c0c78e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c040a8c70 con 0x7f8c0413d5a0 2024-04-26T02:21:35.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c0c78e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c040a8f50 con 0x7f8c040aecc0 2024-04-26T02:21:35.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c0ad2b640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f8c040aecc0 0x7f8c04140860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:41158/0 (socket says 172.21.15.87:41158) 2024-04-26T02:21:35.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c0ad2b640 1 -- 172.21.15.87:0/4076175890 learned_addr learned my addr 172.21.15.87:0/4076175890 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:35.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c0a52a640 1 --1- 172.21.15.87:0/4076175890 >> v1:172.21.15.87:6789/0 conn(0x7f8c040ae8e0 0x7f8c0413ce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:41826/0 (socket says 172.21.15.87:41826) 2024-04-26T02:21:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c09528640 1 -- 172.21.15.87:0/4076175890 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 939127157 0 0) 0x7f8c040a8f50 con 0x7f8c040aecc0 2024-04-26T02:21:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c09528640 1 -- 172.21.15.87:0/4076175890 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bf0003530 con 0x7f8c040aecc0 2024-04-26T02:21:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c09528640 1 -- 172.21.15.87:0/4076175890 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1923474718 0 0) 0x7f8c040a8990 con 0x7f8c040ae8e0 2024-04-26T02:21:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c09528640 1 -- 172.21.15.87:0/4076175890 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c040a8f50 con 0x7f8c040ae8e0 2024-04-26T02:21:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c09528640 1 -- 172.21.15.87:0/4076175890 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2307904727 0 0) 0x7f8bf0003530 con 0x7f8c040aecc0 2024-04-26T02:21:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c09528640 1 -- 172.21.15.87:0/4076175890 >> v1:172.21.15.122:6789/0 conn(0x7f8c0413d5a0 legacy=0x7f8c04140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c09528640 1 -- 172.21.15.87:0/4076175890 >> v1:172.21.15.87:6789/0 conn(0x7f8c040ae8e0 legacy=0x7f8c0413ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c09528640 1 -- 172.21.15.87:0/4076175890 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c0414f7d0 con 0x7f8c040aecc0 2024-04-26T02:21:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.557+0000 7f8c09528640 1 -- 172.21.15.87:0/4076175890 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8bfc0019e0 con 0x7f8c040aecc0 2024-04-26T02:21:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c09528640 1 -- 172.21.15.87:0/4076175890 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8bfc001c00 con 0x7f8c040aecc0 2024-04-26T02:21:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c09528640 1 -- 172.21.15.87:0/4076175890 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8bfc001fd0 con 0x7f8c040aecc0 2024-04-26T02:21:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c0c78e640 1 -- 172.21.15.87:0/4076175890 >> v1:172.21.15.87:6790/0 conn(0x7f8c040aecc0 legacy=0x7f8c04140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c0c78e640 1 -- 172.21.15.87:0/4076175890 shutdown_connections 2024-04-26T02:21:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c0c78e640 1 -- 172.21.15.87:0/4076175890 >> 172.21.15.87:0/4076175890 conn(0x7f8c040b4230 msgr2=0x7f8c0406b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c0c78e640 1 -- 172.21.15.87:0/4076175890 shutdown_connections 2024-04-26T02:21:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c0c78e640 1 -- 172.21.15.87:0/4076175890 wait complete. 2024-04-26T02:21:35.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c0c78e640 1 Processor -- start 2024-04-26T02:21:35.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c0c78e640 1 -- start start 2024-04-26T02:21:35.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c0c78e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c0414d810 con 0x7f8c0413d5a0 2024-04-26T02:21:35.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c0c78e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c0414e6b0 con 0x7f8c040ae8e0 2024-04-26T02:21:35.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c0c78e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c0413fb00 con 0x7f8c040aecc0 2024-04-26T02:21:35.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c09d29640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f8c040aecc0 0x7f8c04070440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:41174/0 (socket says 172.21.15.87:41174) 2024-04-26T02:21:35.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c09d29640 1 -- 172.21.15.87:0/854352883 learned_addr learned my addr 172.21.15.87:0/854352883 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:35.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3446601766 0 0) 0x7f8c0414d810 con 0x7f8c0413d5a0 2024-04-26T02:21:35.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8be8003530 con 0x7f8c0413d5a0 2024-04-26T02:21:35.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3983275214 0 0) 0x7f8c0413fb00 con 0x7f8c040aecc0 2024-04-26T02:21:35.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c0414d810 con 0x7f8c040aecc0 2024-04-26T02:21:35.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2525161725 0 0) 0x7f8c0414d810 con 0x7f8c040aecc0 2024-04-26T02:21:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8c0413fb00 con 0x7f8c040aecc0 2024-04-26T02:21:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8bf40018e0 con 0x7f8c040aecc0 2024-04-26T02:21:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3385635511 0 0) 0x7f8be8003530 con 0x7f8c0413d5a0 2024-04-26T02:21:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8c0414d810 con 0x7f8c0413d5a0 2024-04-26T02:21:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8bfc001f50 con 0x7f8c0413d5a0 2024-04-26T02:21:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2775251699 0 0) 0x7f8c0413fb00 con 0x7f8c040aecc0 2024-04-26T02:21:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 >> v1:172.21.15.122:6789/0 conn(0x7f8c040ae8e0 legacy=0x7f8c0414d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 >> v1:172.21.15.87:6789/0 conn(0x7f8c0413d5a0 legacy=0x7f8c041e2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c04070b50 con 0x7f8c040aecc0 2024-04-26T02:21:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c0c78e640 1 -- 172.21.15.87:0/854352883 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8c04070e20 con 0x7f8c040aecc0 2024-04-26T02:21:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c0c78e640 1 -- 172.21.15.87:0/854352883 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8c0414eff0 con 0x7f8c040aecc0 2024-04-26T02:21:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8bf4003620 con 0x7f8c040aecc0 2024-04-26T02:21:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8bf40039f0 con 0x7f8c040aecc0 2024-04-26T02:21:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.561+0000 7f8c0c78e640 1 -- 172.21.15.87:0/854352883 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bec005090 con 0x7f8c040aecc0 2024-04-26T02:21:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.565+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f8bf4003c10 con 0x7f8c040aecc0 2024-04-26T02:21:35.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.565+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 4090+0+0 (unknown 2211391313 0 0) 0x7f8bf4081000 con 0x7f8c040aecc0 2024-04-26T02:21:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.569+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f8bf4052160 con 0x7f8c040aecc0 2024-04-26T02:21:35.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.749+0000 7f8c0c78e640 1 -- 172.21.15.87:0/854352883 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f8bec005380 con 0x7f8c040aecc0 2024-04-26T02:21:35.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.749+0000 7f8bfaffd640 1 -- 172.21.15.87:0/854352883 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f8bf4051a60 con 0x7f8c040aecc0 2024-04-26T02:21:35.748 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:21:35.748 INFO:teuthology.orchestra.run.smithi087.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T02:21:35.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.753+0000 7f8c0c78e640 1 -- 172.21.15.87:0/854352883 >> v1:172.21.15.122:6816/416193625 conn(0x7f8be8068360 legacy=0x7f8be806a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.753+0000 7f8c0c78e640 1 -- 172.21.15.87:0/854352883 >> v1:172.21.15.87:6790/0 conn(0x7f8c040aecc0 legacy=0x7f8c04070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.753+0000 7f8c0c78e640 1 -- 172.21.15.87:0/854352883 shutdown_connections 2024-04-26T02:21:35.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.753+0000 7f8c0c78e640 1 -- 172.21.15.87:0/854352883 >> 172.21.15.87:0/854352883 conn(0x7f8c040b4230 msgr2=0x7f8c040b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:35.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.753+0000 7f8c0c78e640 1 -- 172.21.15.87:0/854352883 shutdown_connections 2024-04-26T02:21:35.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.753+0000 7f8c0c78e640 1 -- 172.21.15.87:0/854352883 wait complete. 2024-04-26T02:21:35.765 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-26T02:21:35.765 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T02:21:35.775 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T02:21:35.775 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi087.front.sepia.ceph.com 2024-04-26T02:21:35.775 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph osd pool create cephsqlite' 2024-04-26T02:21:35.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.885+0000 7ff8747a9640 1 Processor -- start 2024-04-26T02:21:35.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.885+0000 7ff8747a9640 1 -- start start 2024-04-26T02:21:35.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.885+0000 7ff8747a9640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff86c1430a0 con 0x7ff86c0b4e10 2024-04-26T02:21:35.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.885+0000 7ff8747a9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff86c142190 con 0x7ff86c0a9910 2024-04-26T02:21:35.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.885+0000 7ff8747a9640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff86c13c520 con 0x7ff86c0b27b0 2024-04-26T02:21:35.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.885+0000 7ff871d44640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7ff86c0b27b0 0x7ff86c13be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:41190/0 (socket says 172.21.15.87:41190) 2024-04-26T02:21:35.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.885+0000 7ff871d44640 1 -- 172.21.15.87:0/3936744897 learned_addr learned my addr 172.21.15.87:0/3936744897 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:35.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff871543640 1 -- 172.21.15.87:0/3936744897 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 694451138 0 0) 0x7ff86c1430a0 con 0x7ff86c0b4e10 2024-04-26T02:21:35.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff871543640 1 -- 172.21.15.87:0/3936744897 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff858003530 con 0x7ff86c0b4e10 2024-04-26T02:21:35.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff871543640 1 -- 172.21.15.87:0/3936744897 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 366165789 0 0) 0x7ff86c13c520 con 0x7ff86c0b27b0 2024-04-26T02:21:35.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff871543640 1 -- 172.21.15.87:0/3936744897 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff86c1430a0 con 0x7ff86c0b27b0 2024-04-26T02:21:35.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff872d46640 1 -- 172.21.15.87:0/3936744897 >> v1:172.21.15.122:6789/0 conn(0x7ff86c0a9910 legacy=0x7ff86c0b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T02:21:35.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff872d46640 1 -- 172.21.15.87:0/3936744897 >> v1:172.21.15.122:6789/0 conn(0x7ff86c0a9910 legacy=0x7ff86c0b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T02:21:35.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff872d46640 1 --1- 172.21.15.87:0/3936744897 >> v1:172.21.15.122:6789/0 conn(0x7ff86c0a9910 0x7ff86c0b17b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-04-26T02:21:35.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff871543640 1 -- 172.21.15.87:0/3936744897 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1591680595 0 0) 0x7ff858003530 con 0x7ff86c0b4e10 2024-04-26T02:21:35.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff871543640 1 -- 172.21.15.87:0/3936744897 >> v1:172.21.15.87:6790/0 conn(0x7ff86c0b27b0 legacy=0x7ff86c13be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff871543640 1 -- 172.21.15.87:0/3936744897 >> v1:172.21.15.122:6789/0 conn(0x7ff86c0a9910 legacy=0x7ff86c0b17b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T02:21:35.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff871543640 1 -- 172.21.15.87:0/3936744897 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86c0a9d40 con 0x7ff86c0b4e10 2024-04-26T02:21:35.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff871543640 1 -- 172.21.15.87:0/3936744897 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7ff85c001620 con 0x7ff86c0b4e10 2024-04-26T02:21:35.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff871543640 1 -- 172.21.15.87:0/3936744897 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff85c001940 con 0x7ff86c0b4e10 2024-04-26T02:21:35.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff871543640 1 -- 172.21.15.87:0/3936744897 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7ff85c001d10 con 0x7ff86c0b4e10 2024-04-26T02:21:35.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff8747a9640 1 -- 172.21.15.87:0/3936744897 >> v1:172.21.15.87:6789/0 conn(0x7ff86c0b4e10 legacy=0x7ff86c142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff8747a9640 1 -- 172.21.15.87:0/3936744897 shutdown_connections 2024-04-26T02:21:35.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff8747a9640 1 -- 172.21.15.87:0/3936744897 >> 172.21.15.87:0/3936744897 conn(0x7ff86c072ad0 msgr2=0x7ff86c072ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:35.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff8747a9640 1 -- 172.21.15.87:0/3936744897 shutdown_connections 2024-04-26T02:21:35.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff8747a9640 1 -- 172.21.15.87:0/3936744897 wait complete. 2024-04-26T02:21:35.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff8747a9640 1 Processor -- start 2024-04-26T02:21:35.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff8747a9640 1 -- start start 2024-04-26T02:21:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff8747a9640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff86c1d7be0 con 0x7ff86c0a9910 2024-04-26T02:21:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff8747a9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff86c1d8810 con 0x7ff86c0b27b0 2024-04-26T02:21:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff8747a9640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff86c13d4f0 con 0x7ff86c0b4e10 2024-04-26T02:21:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff872545640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7ff86c0a9910 0x7ff86c0ab920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:41848/0 (socket says 172.21.15.87:41848) 2024-04-26T02:21:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff872545640 1 -- 172.21.15.87:0/445930866 learned_addr learned my addr 172.21.15.87:0/445930866 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff872d46640 1 --1- 172.21.15.87:0/445930866 >> v1:172.21.15.87:6790/0 conn(0x7ff86c0b4e10 0x7ff86c138220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:41192/0 (socket says 172.21.15.87:41192) 2024-04-26T02:21:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff872545640 1 -- 172.21.15.87:0/445930866 >> v1:172.21.15.87:6789/0 conn(0x7ff86c0a9910 legacy=0x7ff86c0ab920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T02:21:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff872545640 1 -- 172.21.15.87:0/445930866 >> v1:172.21.15.87:6789/0 conn(0x7ff86c0a9910 legacy=0x7ff86c0ab920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T02:21:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff872545640 1 --1- 172.21.15.87:0/445930866 >> v1:172.21.15.87:6789/0 conn(0x7ff86c0a9910 0x7ff86c0ab920 :-1 s=OPENED pgs=139 cs=1 l=1).handle_message read tag failed 2024-04-26T02:21:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff872545640 1 --1- 172.21.15.87:0/445930866 >> v1:172.21.15.87:6789/0 conn(0x7ff86c0a9910 0x7ff86c0ab920 :-1 s=OPENED pgs=139 cs=1 l=1).fault on lossy channel, failing 2024-04-26T02:21:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff862ffd640 1 -- 172.21.15.87:0/445930866 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3699053503 0 0) 0x7ff86c13d4f0 con 0x7ff86c0b4e10 2024-04-26T02:21:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff862ffd640 1 -- 172.21.15.87:0/445930866 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff850003530 con 0x7ff86c0b4e10 2024-04-26T02:21:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff862ffd640 1 -- 172.21.15.87:0/445930866 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2077387680 0 0) 0x7ff850003530 con 0x7ff86c0b4e10 2024-04-26T02:21:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff862ffd640 1 -- 172.21.15.87:0/445930866 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff86c13d4f0 con 0x7ff86c0b4e10 2024-04-26T02:21:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff862ffd640 1 -- 172.21.15.87:0/445930866 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7ff864004210 con 0x7ff86c0b4e10 2024-04-26T02:21:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff862ffd640 1 -- 172.21.15.87:0/445930866 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2707144699 0 0) 0x7ff86c13d4f0 con 0x7ff86c0b4e10 2024-04-26T02:21:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff862ffd640 1 -- 172.21.15.87:0/445930866 >> v1:172.21.15.122:6789/0 conn(0x7ff86c0b27b0 legacy=0x7ff86c1d8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff862ffd640 1 -- 172.21.15.87:0/445930866 >> v1:172.21.15.87:6789/0 conn(0x7ff86c0a9910 legacy=0x7ff86c0ab920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T02:21:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff862ffd640 1 -- 172.21.15.87:0/445930866 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86c13d6c0 con 0x7ff86c0b4e10 2024-04-26T02:21:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff8747a9640 1 -- 172.21.15.87:0/445930866 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff86c138930 con 0x7ff86c0b4e10 2024-04-26T02:21:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff8747a9640 1 -- 172.21.15.87:0/445930866 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff86c0ac650 con 0x7ff86c0b4e10 2024-04-26T02:21:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff862ffd640 1 -- 172.21.15.87:0/445930866 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff864001260 con 0x7ff86c0b4e10 2024-04-26T02:21:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.889+0000 7ff862ffd640 1 -- 172.21.15.87:0/445930866 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7ff864001630 con 0x7ff86c0b4e10 2024-04-26T02:21:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.893+0000 7ff8747a9640 1 -- 172.21.15.87:0/445930866 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff854005090 con 0x7ff86c0b4e10 2024-04-26T02:21:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.893+0000 7ff862ffd640 1 -- 172.21.15.87:0/445930866 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7ff864001850 con 0x7ff86c0b4e10 2024-04-26T02:21:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.893+0000 7ff862ffd640 1 -- 172.21.15.87:0/445930866 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 4090+0+0 (unknown 2211391313 0 0) 0x7ff864081350 con 0x7ff86c0b4e10 2024-04-26T02:21:35.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:35.897+0000 7ff862ffd640 1 -- 172.21.15.87:0/445930866 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7ff864052370 con 0x7ff86c0b4e10 2024-04-26T02:21:36.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.045+0000 7ff8747a9640 1 -- 172.21.15.87:0/445930866 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "cephsqlite"} v 0) -- 0x7ff854005380 con 0x7ff86c0b4e10 2024-04-26T02:21:36.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.469+0000 7ff862ffd640 1 -- 172.21.15.87:0/445930866 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephsqlite"}]=0 pool 'cephsqlite' created v20) ==== 110+0+0 (unknown 2615499700 0 0) 0x7ff864051c70 con 0x7ff86c0b4e10 2024-04-26T02:21:36.469 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'cephsqlite' created 2024-04-26T02:21:36.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.477+0000 7ff8747a9640 1 -- 172.21.15.87:0/445930866 >> v1:172.21.15.122:6816/416193625 conn(0x7ff8500682d0 legacy=0x7ff85006a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:36.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.477+0000 7ff8747a9640 1 -- 172.21.15.87:0/445930866 >> v1:172.21.15.87:6790/0 conn(0x7ff86c0b4e10 legacy=0x7ff86c138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:36.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.477+0000 7ff8747a9640 1 -- 172.21.15.87:0/445930866 shutdown_connections 2024-04-26T02:21:36.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.477+0000 7ff8747a9640 1 -- 172.21.15.87:0/445930866 >> 172.21.15.87:0/445930866 conn(0x7ff86c072ad0 msgr2=0x7ff86c141660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:36.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.477+0000 7ff8747a9640 1 -- 172.21.15.87:0/445930866 shutdown_connections 2024-04-26T02:21:36.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.477+0000 7ff8747a9640 1 -- 172.21.15.87:0/445930866 wait complete. 2024-04-26T02:21:36.494 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph auth get-or-create client.libcephsqlite mon '"'"'profile simple-rados-client-with-blocklist'"'"' osd '"'"'allow rwx pool=cephsqlite'"'"' >> /etc/ceph/ceph.keyring' 2024-04-26T02:21:36.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.621+0000 7f9fa0744640 1 Processor -- start 2024-04-26T02:21:36.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.621+0000 7f9fa0744640 1 -- start start 2024-04-26T02:21:36.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9fa0744640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f980b4740 con 0x7f9f980afc20 2024-04-26T02:21:36.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9fa0744640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f9813fda0 con 0x7f9f980b4310 2024-04-26T02:21:36.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9fa0744640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f980b1080 con 0x7f9f9806d2b0 2024-04-26T02:21:36.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9dcdf640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f9f980afc20 0x7f9f980b14a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:41872/0 (socket says 172.21.15.87:41872) 2024-04-26T02:21:36.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9dcdf640 1 -- 172.21.15.87:0/599559820 learned_addr learned my addr 172.21.15.87:0/599559820 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:36.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9e4e0640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f9f9806d2b0 0x7f9f9813f690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:41208/0 (socket says 172.21.15.87:41208) 2024-04-26T02:21:36.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9d4de640 1 -- 172.21.15.87:0/599559820 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1777925254 0 0) 0x7f9f88001120 con 0x7f9f980afc20 2024-04-26T02:21:36.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9d4de640 1 -- 172.21.15.87:0/599559820 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f84003530 con 0x7f9f980afc20 2024-04-26T02:21:36.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9d4de640 1 -- 172.21.15.87:0/599559820 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771487537 0 0) 0x7f9f980b1080 con 0x7f9f9806d2b0 2024-04-26T02:21:36.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9d4de640 1 -- 172.21.15.87:0/599559820 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f88001120 con 0x7f9f9806d2b0 2024-04-26T02:21:36.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9d4de640 1 -- 172.21.15.87:0/599559820 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2131284118 0 0) 0x7f9f84003530 con 0x7f9f980afc20 2024-04-26T02:21:36.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9d4de640 1 -- 172.21.15.87:0/599559820 >> v1:172.21.15.87:6790/0 conn(0x7f9f9806d2b0 legacy=0x7f9f9813f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:36.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9d4de640 1 -- 172.21.15.87:0/599559820 >> v1:172.21.15.122:6789/0 conn(0x7f9f980b4310 legacy=0x7f9f9814da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:36.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9d4de640 1 -- 172.21.15.87:0/599559820 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f980af700 con 0x7f9f980afc20 2024-04-26T02:21:36.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9d4de640 1 -- 172.21.15.87:0/599559820 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9f880017f0 con 0x7f9f980afc20 2024-04-26T02:21:36.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9d4de640 1 -- 172.21.15.87:0/599559820 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9f88001ad0 con 0x7f9f980afc20 2024-04-26T02:21:36.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9d4de640 1 -- 172.21.15.87:0/599559820 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9f880048b0 con 0x7f9f980afc20 2024-04-26T02:21:36.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9fa0744640 1 -- 172.21.15.87:0/599559820 >> v1:172.21.15.87:6789/0 conn(0x7f9f980afc20 legacy=0x7f9f980b14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:36.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9fa0744640 1 -- 172.21.15.87:0/599559820 shutdown_connections 2024-04-26T02:21:36.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9fa0744640 1 -- 172.21.15.87:0/599559820 >> 172.21.15.87:0/599559820 conn(0x7f9f980b7ff0 msgr2=0x7f9f980b8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:36.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9fa0744640 1 -- 172.21.15.87:0/599559820 shutdown_connections 2024-04-26T02:21:36.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9fa0744640 1 -- 172.21.15.87:0/599559820 wait complete. 2024-04-26T02:21:36.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9fa0744640 1 Processor -- start 2024-04-26T02:21:36.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9fa0744640 1 -- start start 2024-04-26T02:21:36.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9fa0744640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f980a98e0 con 0x7f9f9806d2b0 2024-04-26T02:21:36.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9fa0744640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f980a9bc0 con 0x7f9f980afc20 2024-04-26T02:21:36.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9fa0744640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f980a9ea0 con 0x7f9f980b4310 2024-04-26T02:21:36.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9ece1640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f9f980b4310 0x7f9f981e2490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:41224/0 (socket says 172.21.15.87:41224) 2024-04-26T02:21:36.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9ece1640 1 -- 172.21.15.87:0/929139283 learned_addr learned my addr 172.21.15.87:0/929139283 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:21:36.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 813432004 0 0) 0x7f9f980a9ea0 con 0x7f9f980b4310 2024-04-26T02:21:36.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f7c003530 con 0x7f9f980b4310 2024-04-26T02:21:36.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1793234345 0 0) 0x7f9f980a98e0 con 0x7f9f9806d2b0 2024-04-26T02:21:36.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f980a9ea0 con 0x7f9f9806d2b0 2024-04-26T02:21:36.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9e4e0640 1 -- 172.21.15.87:0/929139283 >> v1:172.21.15.87:6789/0 conn(0x7f9f9806d2b0 legacy=0x7f9f980aa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T02:21:36.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9e4e0640 1 -- 172.21.15.87:0/929139283 >> v1:172.21.15.87:6789/0 conn(0x7f9f9806d2b0 legacy=0x7f9f980aa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T02:21:36.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9e4e0640 1 --1- 172.21.15.87:0/929139283 >> v1:172.21.15.87:6789/0 conn(0x7f9f9806d2b0 0x7f9f980aa2c0 :-1 s=OPENED pgs=142 cs=1 l=1).handle_message read tag failed 2024-04-26T02:21:36.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f9e4e0640 1 --1- 172.21.15.87:0/929139283 >> v1:172.21.15.87:6789/0 conn(0x7f9f9806d2b0 0x7f9f980aa2c0 :-1 s=OPENED pgs=142 cs=1 l=1).fault on lossy channel, failing 2024-04-26T02:21:36.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4139614317 0 0) 0x7f9f7c003530 con 0x7f9f980b4310 2024-04-26T02:21:36.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f980a98e0 con 0x7f9f980b4310 2024-04-26T02:21:36.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.625+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9f90001c20 con 0x7f9f980b4310 2024-04-26T02:21:36.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.629+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 478507023 0 0) 0x7f9f980a98e0 con 0x7f9f980b4310 2024-04-26T02:21:36.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.629+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 >> v1:172.21.15.122:6789/0 conn(0x7f9f980afc20 legacy=0x7f9f98145aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:36.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.629+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 >> v1:172.21.15.87:6789/0 conn(0x7f9f9806d2b0 legacy=0x7f9f980aa2c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T02:21:36.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.629+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f98145580 con 0x7f9f980b4310 2024-04-26T02:21:36.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.629+0000 7f9fa0744640 1 -- 172.21.15.87:0/929139283 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9f98144b20 con 0x7f9f980b4310 2024-04-26T02:21:36.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.629+0000 7f9fa0744640 1 -- 172.21.15.87:0/929139283 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9f9813c2a0 con 0x7f9f980b4310 2024-04-26T02:21:36.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.629+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9f900039a0 con 0x7f9f980b4310 2024-04-26T02:21:36.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.629+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9f90003d70 con 0x7f9f980b4310 2024-04-26T02:21:36.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.629+0000 7f9fa0744640 1 -- 172.21.15.87:0/929139283 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f80005090 con 0x7f9f980b4310 2024-04-26T02:21:36.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.629+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f9f90003f10 con 0x7f9f980b4310 2024-04-26T02:21:36.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.629+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 4439+0+0 (unknown 275360379 0 0) 0x7f9f900814a0 con 0x7f9f980b4310 2024-04-26T02:21:36.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.633+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f9f90052360 con 0x7f9f980b4310 2024-04-26T02:21:36.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.833+0000 7f9fa0744640 1 -- 172.21.15.87:0/929139283 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.libcephsqlite", "caps": ["mon", "profile simple-rados-client-with-blocklist", "osd", "allow rwx pool=cephsqlite"]} v 0) -- 0x7f9f80005380 con 0x7f9f980b4310 2024-04-26T02:21:36.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.841+0000 7f9f8effd640 1 -- 172.21.15.87:0/929139283 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.libcephsqlite", "caps": ["mon", "profile simple-rados-client-with-blocklist", "osd", "allow rwx pool=cephsqlite"]}]=0 v13) ==== 199+0+71 (unknown 96269769 0 3466938963) 0x7f9f90051c60 con 0x7f9f980b4310 2024-04-26T02:21:36.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.845+0000 7f9fa0744640 1 -- 172.21.15.87:0/929139283 >> v1:172.21.15.122:6816/416193625 conn(0x7f9f7c0682f0 legacy=0x7f9f7c06a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:36.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.845+0000 7f9fa0744640 1 -- 172.21.15.87:0/929139283 >> v1:172.21.15.87:6790/0 conn(0x7f9f980b4310 legacy=0x7f9f981e2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:21:36.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.845+0000 7f9fa0744640 1 -- 172.21.15.87:0/929139283 shutdown_connections 2024-04-26T02:21:36.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.845+0000 7f9fa0744640 1 -- 172.21.15.87:0/929139283 >> 172.21.15.87:0/929139283 conn(0x7f9f980b7ff0 msgr2=0x7f9f9813f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:21:36.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.845+0000 7f9fa0744640 1 -- 172.21.15.87:0/929139283 shutdown_connections 2024-04-26T02:21:36.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:21:36.845+0000 7f9fa0744640 1 -- 172.21.15.87:0/929139283 wait complete. 2024-04-26T02:21:36.858 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T02:21:36.866 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T02:21:36.867 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi087.front.sepia.ceph.com 2024-04-26T02:21:36.867 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_libcephsqlite --id libcephsqlite --no-log-to-stderr' 2024-04-26T02:21:36.901 INFO:teuthology.orchestra.run.smithi087.stdout:[==========] Running 20 tests from 1 test suite. 2024-04-26T02:21:36.901 INFO:teuthology.orchestra.run.smithi087.stdout:[----------] Global test environment set-up. 2024-04-26T02:21:36.901 INFO:teuthology.orchestra.run.smithi087.stdout:[----------] 20 tests from CephSQLiteTest 2024-04-26T02:21:41.908 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.Create 2024-04-26T02:21:41.922 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/502ffa89-7dc6-483e-95e9-ed39bc6c8d75.db?vfs=ceph 2024-04-26T02:21:41.922 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:21:41.931 INFO:teuthology.orchestra.run.smithi087.stdout:CREATE TABLE foo (a INT); 2024-04-26T02:21:41.957 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.Create (50 ms) 2024-04-26T02:21:41.957 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.InsertBulk4096 2024-04-26T02:21:41.968 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/eafc23e4-bdb3-4fa5-aeac-8e791740678f.db?vfs=ceph 2024-04-26T02:21:41.969 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:21:41.976 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA page_size = 4096;CREATE TABLE foo (a INT);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO foo (a) SELECT RANDOM() FROM c LIMIT 1000000;PRAGMA page_size; 2024-04-26T02:21:44.455 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.InsertBulk4096 (2497 ms) 2024-04-26T02:21:44.455 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.InsertBulk 2024-04-26T02:21:44.469 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/412c5875-a1d0-4949-92a8-2491da53d088.db?vfs=ceph 2024-04-26T02:21:44.469 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:21:44.476 INFO:teuthology.orchestra.run.smithi087.stdout:CREATE TABLE foo (a INT);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO foo (a) SELECT RANDOM() FROM c LIMIT 1000000; 2024-04-26T02:21:45.070 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.InsertBulk (616 ms) 2024-04-26T02:21:45.071 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.UpdateBulk 2024-04-26T02:21:45.082 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/72d57a09-9908-46d7-b745-c2c36037341f.db?vfs=ceph 2024-04-26T02:21:45.083 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:21:45.087 INFO:teuthology.orchestra.run.smithi087.stdout:CREATE TABLE foo (a INT);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO foo (a) SELECT x FROM c LIMIT 1000000;SELECT SUM(a) FROM foo;UPDATE foo SET a = a+a;SELECT SUM(a) FROM foo; 2024-04-26T02:21:46.397 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.UpdateBulk (1326 ms) 2024-04-26T02:21:46.397 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.InsertRate 2024-04-26T02:21:46.408 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/87dbc142-ebdd-4811-9a68-6e69eaf5b409.db?vfs=ceph 2024-04-26T02:21:46.408 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:21:46.418 INFO:teuthology.orchestra.run.smithi087.stdout:CREATE TABLE foo (a INT);INSERT INTO foo (a) VALUES (RANDOM()); 2024-04-26T02:21:49.826 INFO:teuthology.orchestra.run.smithi087.stdout:transactions per second: 29.6563 2024-04-26T02:21:49.827 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.InsertRate (3431 ms) 2024-04-26T02:21:49.827 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.DatabaseShrink 2024-04-26T02:21:49.840 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/0afacffc-be80-4c26-a2b1-c4a7c4c8f635.db?vfs=ceph 2024-04-26T02:21:49.841 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:21:49.847 INFO:teuthology.orchestra.run.smithi087.stdout:CREATE TABLE foo (a INT);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO foo (a) SELECT x FROM c LIMIT 1000000;DELETE FROM foo WHERE RANDOM()%4 < 3;VACUUM; 2024-04-26T02:21:51.573 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.DatabaseShrink (1746 ms) 2024-04-26T02:21:51.574 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.InsertExclusiveRate 2024-04-26T02:21:51.586 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/07ff0298-e256-43c6-92a2-77a4be107ab0.db?vfs=ceph 2024-04-26T02:21:51.586 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:21:51.593 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA locking_mode=EXCLUSIVE;CREATE TABLE foo (a INT);INSERT INTO foo (a) VALUES (RANDOM()); 2024-04-26T02:21:53.159 INFO:teuthology.orchestra.run.smithi087.stdout:transactions per second: 64.774 2024-04-26T02:21:53.164 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.InsertExclusiveRate (1590 ms) 2024-04-26T02:21:53.164 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.InsertExclusiveWALRate 2024-04-26T02:21:53.176 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/31732e2d-48b3-476b-a162-909ec75b0362.db?vfs=ceph 2024-04-26T02:21:53.177 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:21:53.184 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA locking_mode=EXCLUSIVE;PRAGMA journal_mode=WAL;CREATE TABLE foo (a INT);INSERT INTO foo (a) VALUES (RANDOM()); 2024-04-26T02:21:53.743 INFO:teuthology.orchestra.run.smithi087.stdout:transactions per second: 198.44 2024-04-26T02:21:53.777 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.InsertExclusiveWALRate (614 ms) 2024-04-26T02:21:53.778 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.WALTransactionSync 2024-04-26T02:21:53.793 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/5776e835-ce48-404e-963a-4dbd0ac37370.db?vfs=ceph 2024-04-26T02:21:53.793 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:21:53.799 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA locking_mode=EXCLUSIVE;PRAGMA journal_mode=WAL;CREATE TABLE foo (a INT);INSERT INTO perf (v) VALUES (ceph_perf());BEGIN TRANSACTION;INSERT INTO foo (a) VALUES (RANDOM());END TRANSACTION;INSERT INTO perf (v) VALUES (ceph_perf());SELECT a.atom-b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_vfs.opf_sync.avgcount' AND b.fullkey = '$.libcephsqlite_vfs.opf_sync.avgcount'; 2024-04-26T02:21:53.888 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.WALTransactionSync (110 ms) 2024-04-26T02:21:53.888 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.PersistTransactionSync 2024-04-26T02:21:53.898 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/a5156051-2072-4080-bf56-9ade63de705d.db?vfs=ceph 2024-04-26T02:21:53.898 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:21:53.903 INFO:teuthology.orchestra.run.smithi087.stdout:BEGIN TRANSACTION;CREATE TABLE foo (a INT);INSERT INTO foo (a) VALUES (RANDOM());END TRANSACTION;INSERT INTO perf (v) VALUES (ceph_perf());SELECT a.atom-b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_vfs.opf_sync.avgcount' AND b.fullkey = '$.libcephsqlite_vfs.opf_sync.avgcount'; 2024-04-26T02:21:53.932 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.PersistTransactionSync (45 ms) 2024-04-26T02:21:53.933 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.InsertExclusiveLock 2024-04-26T02:21:53.942 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/7dbf606e-78d5-4742-96fc-e05423c961fe.db?vfs=ceph 2024-04-26T02:21:53.942 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:21:53.948 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA locking_mode=EXCLUSIVE;CREATE TABLE foo (a INT);INSERT INTO foo (a) VALUES (RANDOM());INSERT INTO perf (v) VALUES (ceph_perf());SELECT a.atom, b.atom, a.atom-b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_vfs.opf_lock.avgcount' AND b.fullkey = '$.libcephsqlite_vfs.opf_lock.avgcount';SELECT a.atom, b.atom, a.atom-b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_striper.lock' AND b.fullkey = '$.libcephsqlite_striper.lock'; 2024-04-26T02:21:53.983 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.InsertExclusiveLock (50 ms) 2024-04-26T02:21:53.983 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.TransactionSizeUpdate 2024-04-26T02:21:53.994 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/539cdece-d2e1-4339-8a89-62b4bac47496.db?vfs=ceph 2024-04-26T02:21:53.994 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:21:54.000 INFO:teuthology.orchestra.run.smithi087.stdout:BEGIN TRANSACTION;CREATE TABLE foo (a INT);INSERT INTO foo (a) VALUES (RANDOM());END TRANSACTION;INSERT INTO perf (v) VALUES (ceph_perf());SELECT a.atom, b.atom, a.atom-b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_striper.update_size' AND b.fullkey = '$.libcephsqlite_striper.update_size'; 2024-04-26T02:21:54.029 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.TransactionSizeUpdate (47 ms) 2024-04-26T02:21:54.030 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.AllocatedGrowth 2024-04-26T02:21:54.042 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/58d77321-efb5-4ee0-89e6-e9d701aeb645.db?vfs=ceph 2024-04-26T02:21:54.042 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:21:54.049 INFO:teuthology.orchestra.run.smithi087.stdout:CREATE TABLE foo (a BLOB);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO foo (a) SELECT RANDOMBLOB(1<<20) FROM c LIMIT 1024;INSERT INTO perf (v) VALUES (ceph_perf());SELECT a.atom, b.atom, a.atom-b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_striper.update_allocated' AND b.fullkey = '$.libcephsqlite_striper.update_allocated'; 2024-04-26T02:22:01.670 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.AllocatedGrowth (7639 ms) 2024-04-26T02:22:01.671 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.DeleteBulk 2024-04-26T02:22:01.679 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/5670e013-10f7-4613-99ae-ff059b86bc9a.db?vfs=ceph 2024-04-26T02:22:01.679 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:22:01.683 INFO:teuthology.orchestra.run.smithi087.stdout:CREATE TABLE foo (a INT);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO foo (a) SELECT x FROM c LIMIT 1000000;DELETE FROM foo WHERE RANDOM()%2 == 0; 2024-04-26T02:22:02.639 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.DeleteBulk (969 ms) 2024-04-26T02:22:02.639 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.DropMassive 2024-04-26T02:22:02.647 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/8178a872-9d3b-4511-8a5d-e5488ffcc15d.db?vfs=ceph 2024-04-26T02:22:02.647 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:22:02.652 INFO:teuthology.orchestra.run.smithi087.stdout:CREATE TABLE foo (a BLOB);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO foo (a) SELECT RANDOMBLOB(1<<20) FROM c LIMIT 1024;DROP TABLE foo;VACUUM;INSERT INTO perf (v) VALUES (ceph_perf());SELECT a.atom, b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_striper.shrink' AND b.fullkey = '$.libcephsqlite_striper.shrink';SELECT a.atom-b.atom FROM p AS a, p AS b WHERE a.i = ? AND b.i = ? AND a.fullkey = '$.libcephsqlite_striper.shrink_bytes' AND b.fullkey = '$.libcephsqlite_striper.shrink_bytes'; 2024-04-26T02:22:49.172 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.DropMassive (46530 ms) 2024-04-26T02:22:49.173 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.InsertMassiveVerify 2024-04-26T02:22:49.181 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/12be9dd2-0bd1-4edc-88a4-470f6dc90094.db?vfs=ceph 2024-04-26T02:22:49.181 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:22:49.185 INFO:teuthology.orchestra.run.smithi087.stdout:CREATE TABLE foo (a BLOB);CREATE TEMPORARY TABLE bar (a BLOB);WITH RECURSIVE c(x) AS ( VALUES(1) UNION ALL SELECT x+1 FROM c )INSERT INTO bar (a) SELECT RANDOMBLOB(1<<20) FROM c LIMIT 1024;SELECT a FROM bar;INSERT INTO foo (a) SELECT a FROM bar;SELECT a FROM foo; 2024-04-26T02:23:03.944 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.InsertMassiveVerify (14771 ms) 2024-04-26T02:23:03.944 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.PerfValid 2024-04-26T02:23:03.953 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/9e83efa1-0e4f-4c42-9893-9618ae482a4e.db?vfs=ceph 2024-04-26T02:23:03.953 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:23:03.959 INFO:teuthology.orchestra.run.smithi087.stdout:SELECT json_valid(ceph_perf()); 2024-04-26T02:23:03.959 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.PerfValid (16 ms) 2024-04-26T02:23:03.959 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.StatusValid 2024-04-26T02:23:03.972 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/8e975000-a0db-4f0a-ba6d-04ea9295fac9.db?vfs=ceph 2024-04-26T02:23:03.972 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:23:03.982 INFO:teuthology.orchestra.run.smithi087.stdout:SELECT json_valid(ceph_status()); 2024-04-26T02:23:03.982 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.StatusValid (22 ms) 2024-04-26T02:23:03.982 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.CurrentTime 2024-04-26T02:23:03.995 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/da32c9b1-f8dc-4500-b09b-f78bb4bbff66.db?vfs=ceph 2024-04-26T02:23:03.995 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:23:04.008 INFO:teuthology.orchestra.run.smithi087.stdout:SELECT strftime('%s', 'now'); 2024-04-26T02:23:04.008 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.CurrentTime (26 ms) 2024-04-26T02:23:04.008 INFO:teuthology.orchestra.run.smithi087.stdout:[ RUN ] CephSQLiteTest.StatusFields 2024-04-26T02:23:04.023 INFO:teuthology.orchestra.run.smithi087.stdout:using database: file:cephsqlite:/68977019-4753-4e11-854e-42c8019b5931.db?vfs=ceph 2024-04-26T02:23:04.023 INFO:teuthology.orchestra.run.smithi087.stdout:PRAGMA journal_mode = PERSIST;PRAGMA page_size = 65536;PRAGMA cache_size = 32768;PRAGMA temp_store = memory;CREATE TEMPORARY TABLE perf (i INTEGER PRIMARY KEY, v TEXT);CREATE TEMPORARY VIEW p AS SELECT perf.i, J.* FROM perf, json_tree(perf.v) AS J;INSERT INTO perf (v) VALUES (ceph_perf()); 2024-04-26T02:23:04.035 INFO:teuthology.orchestra.run.smithi087.stdout:SELECT json_extract(ceph_status(), '$.addr');SELECT json_extract(ceph_status(), '$.id'); 2024-04-26T02:23:04.035 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:0/2816725008 2024-04-26T02:23:04.035 INFO:teuthology.orchestra.run.smithi087.stdout:4424 2024-04-26T02:23:04.035 INFO:teuthology.orchestra.run.smithi087.stdout:[ OK ] CephSQLiteTest.StatusFields (27 ms) 2024-04-26T02:23:04.035 INFO:teuthology.orchestra.run.smithi087.stdout:[----------] 20 tests from CephSQLiteTest (82123 ms total) 2024-04-26T02:23:04.035 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:23:04.035 INFO:teuthology.orchestra.run.smithi087.stdout:[----------] Global test environment tear-down 2024-04-26T02:23:04.036 INFO:teuthology.orchestra.run.smithi087.stdout:[==========] 20 tests from 1 test suite ran. (87130 ms total) 2024-04-26T02:23:04.036 INFO:teuthology.orchestra.run.smithi087.stdout:[ PASSED ] 20 tests. 2024-04-26T02:23:04.078 INFO:teuthology.run_tasks:Running task workunit... 2024-04-26T02:23:04.092 INFO:tasks.workunit:Pulling workunits from ref ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T02:23:04.092 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-26T02:23:04.093 DEBUG:teuthology.orchestra.run.smithi087:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-26T02:23:04.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:23:04.127 INFO:teuthology.orchestra.run.smithi087.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-26T02:23:04.127 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-26T02:23:04.174 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-26T02:23:04.174 DEBUG:teuthology.orchestra.run.smithi087:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-26T02:23:04.222 INFO:tasks.workunit:timeout=3h 2024-04-26T02:23:04.223 INFO:tasks.workunit:cleanup=True 2024-04-26T02:23:04.223 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T02:23:04.272 INFO:tasks.workunit.client.0.smithi087.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-26T02:23:51.492 INFO:tasks.workunit.client.0.smithi087.stderr:Note: switching to 'ddeaa91797339a19172c3036ff48cca58c12f448'. 2024-04-26T02:23:51.493 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-26T02:23:51.493 INFO:tasks.workunit.client.0.smithi087.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-26T02:23:51.493 INFO:tasks.workunit.client.0.smithi087.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-26T02:23:51.493 INFO:tasks.workunit.client.0.smithi087.stderr:state without impacting any branches by switching back to a branch. 2024-04-26T02:23:51.493 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-26T02:23:51.493 INFO:tasks.workunit.client.0.smithi087.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-26T02:23:51.493 INFO:tasks.workunit.client.0.smithi087.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-26T02:23:51.493 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-26T02:23:51.494 INFO:tasks.workunit.client.0.smithi087.stderr: git switch -c 2024-04-26T02:23:51.494 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-26T02:23:51.494 INFO:tasks.workunit.client.0.smithi087.stderr:Or undo this operation with: 2024-04-26T02:23:51.494 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-26T02:23:51.494 INFO:tasks.workunit.client.0.smithi087.stderr: git switch - 2024-04-26T02:23:51.494 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-26T02:23:51.494 INFO:tasks.workunit.client.0.smithi087.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-26T02:23:51.494 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-26T02:23:51.494 INFO:tasks.workunit.client.0.smithi087.stderr:HEAD is now at ddeaa917973 Merge branch 'wip-rf-56389-to-squid' of https://github.com/ronen-fr/ceph into wip-yuri2-testing-2024-04-24-0914-squid 2024-04-26T02:23:51.499 DEBUG:teuthology.orchestra.run.smithi087:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-26T02:23:51.549 INFO:tasks.workunit.client.0.smithi087.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-26T02:23:51.551 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-26T02:23:51.551 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-26T02:23:52.327 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-26T02:23:52.407 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-26T02:23:52.461 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-26T02:23:52.462 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-26T02:23:52.462 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-26T02:23:52.507 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-26T02:23:52.511 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-26T02:23:52.511 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-26T02:23:52.563 INFO:tasks.workunit:Running workunits matching rados/test_libcephsqlite.sh on client.0... 2024-04-26T02:23:52.564 INFO:tasks.workunit:Running workunit rados/test_libcephsqlite.sh... 2024-04-26T02:23:52.564 DEBUG:teuthology.orchestra.run.smithi087:workunit test rados/test_libcephsqlite.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ddeaa91797339a19172c3036ff48cca58c12f448 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CEPH_ARGS='--id libcephsqlite --no-log-to-stderr' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_libcephsqlite.sh cephsqlite 2024-04-26T02:23:52.615 INFO:tasks.workunit.client.0.smithi087.stderr:+ pool=cephsqlite 2024-04-26T02:23:52.616 INFO:tasks.workunit.client.0.smithi087.stderr:++ basename /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_libcephsqlite.sh 2024-04-26T02:23:52.616 INFO:tasks.workunit.client.0.smithi087.stderr:+ ns=test_libcephsqlite.sh 2024-04-26T02:23:52.616 INFO:tasks.workunit.client.0.smithi087.stderr:+ striper rm baz.db 2024-04-26T02:23:52.616 INFO:tasks.workunit.client.0.smithi087.stderr:+ rados --pool=cephsqlite --namespace=test_libcephsqlite.sh --striper rm baz.db 2024-04-26T02:23:52.674 INFO:tasks.workunit.client.0.smithi087.stderr:+ sqlite 2024-04-26T02:23:52.674 INFO:tasks.workunit.client.0.smithi087.stderr:+ background= 2024-04-26T02:23:52.675 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' '' = b ']' 2024-04-26T02:23:52.675 INFO:tasks.workunit.client.0.smithi087.stderr:++ cat 2024-04-26T02:23:52.676 INFO:tasks.workunit.client.0.smithi087.stderr:+ a='create table if not exists foo (a INT); 2024-04-26T02:23:52.676 INFO:tasks.workunit.client.0.smithi087.stderr:insert into foo (a) values (RANDOM()); 2024-04-26T02:23:52.676 INFO:tasks.workunit.client.0.smithi087.stderr:drop table foo;' 2024-04-26T02:23:52.676 INFO:tasks.workunit.client.0.smithi087.stderr:+ printf %s 'create table if not exists foo (a INT); 2024-04-26T02:23:52.676 INFO:tasks.workunit.client.0.smithi087.stderr:insert into foo (a) values (RANDOM()); 2024-04-26T02:23:52.676 INFO:tasks.workunit.client.0.smithi087.stderr:drop table foo;' 2024-04-26T02:23:52.676 INFO:tasks.workunit.client.0.smithi087.stderr:create table if not exists foo (a INT); 2024-04-26T02:23:52.676 INFO:tasks.workunit.client.0.smithi087.stderr:insert into foo (a) values (RANDOM()); 2024-04-26T02:23:52.676 INFO:tasks.workunit.client.0.smithi087.stderr:drop table foo;+ '[' '' '!=' b ']' 2024-04-26T02:23:52.676 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 2024-04-26T02:23:52.676 INFO:tasks.workunit.client.0.smithi087.stderr:+ sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd '.open file:///cephsqlite:test_libcephsqlite.sh/baz.db?vfs=ceph' -cmd '.output stdout' 2024-04-26T02:23:53.033 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-26T02:23:53.033 INFO:tasks.workunit.client.0.smithi087.stderr:real 0m0.359s 2024-04-26T02:23:53.033 INFO:tasks.workunit.client.0.smithi087.stderr:user 0m0.074s 2024-04-26T02:23:53.034 INFO:tasks.workunit.client.0.smithi087.stderr:sys 0m0.011s 2024-04-26T02:23:53.034 INFO:tasks.workunit.client.0.smithi087.stderr:+ striper stat baz.db 2024-04-26T02:23:53.034 INFO:tasks.workunit.client.0.smithi087.stderr:+ rados --pool=cephsqlite --namespace=test_libcephsqlite.sh --striper stat baz.db 2024-04-26T02:23:53.075 INFO:tasks.workunit.client.0.smithi087.stdout:cephsqlite/baz.db mtime 2024-04-26T02:23:53.000000+0000, size 8192 2024-04-26T02:23:53.078 INFO:tasks.workunit.client.0.smithi087.stderr:+ striper rm baz.db 2024-04-26T02:23:53.078 INFO:tasks.workunit.client.0.smithi087.stderr:+ rados --pool=cephsqlite --namespace=test_libcephsqlite.sh --striper rm baz.db 2024-04-26T02:23:53.130 INFO:tasks.workunit.client.0.smithi087.stderr:+ sqlite 2024-04-26T02:23:53.130 INFO:tasks.workunit.client.0.smithi087.stderr:++ repeat 10 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:53.130 INFO:tasks.workunit.client.0.smithi087.stderr:++ n=10 2024-04-26T02:23:53.131 INFO:tasks.workunit.client.0.smithi087.stderr:++ shift 2024-04-26T02:23:53.131 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i = 0 )) 2024-04-26T02:23:53.131 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 10 )) 2024-04-26T02:23:53.131 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:53.131 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:53.131 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 10 )) 2024-04-26T02:23:53.131 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:53.131 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:53.131 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 10 )) 2024-04-26T02:23:53.131 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:53.131 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:53.131 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 10 )) 2024-04-26T02:23:53.131 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:53.131 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:53.132 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 10 )) 2024-04-26T02:23:53.132 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:53.132 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:53.133 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 10 )) 2024-04-26T02:23:53.133 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:53.133 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:53.133 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 10 )) 2024-04-26T02:23:53.133 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:53.133 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:53.133 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 10 )) 2024-04-26T02:23:53.133 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:53.134 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:53.134 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 10 )) 2024-04-26T02:23:53.134 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:53.134 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:53.134 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 10 )) 2024-04-26T02:23:53.134 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:53.134 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:53.134 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 10 )) 2024-04-26T02:23:53.134 INFO:tasks.workunit.client.0.smithi087.stderr:+ background= 2024-04-26T02:23:53.134 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' '' = b ']' 2024-04-26T02:23:53.134 INFO:tasks.workunit.client.0.smithi087.stderr:++ cat 2024-04-26T02:23:53.134 INFO:tasks.workunit.client.0.smithi087.stderr:+ a='CREATE TABLE IF NOT EXISTS rand(text BLOB NOT NULL); 2024-04-26T02:23:53.134 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.134 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.135 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.135 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.135 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.135 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.135 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.135 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.135 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.135 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.135 INFO:tasks.workunit.client.0.smithi087.stderr:SELECT LENGTH(text) FROM rand; 2024-04-26T02:23:53.135 INFO:tasks.workunit.client.0.smithi087.stderr:DROP TABLE rand;' 2024-04-26T02:23:53.135 INFO:tasks.workunit.client.0.smithi087.stderr:+ printf %s 'CREATE TABLE IF NOT EXISTS rand(text BLOB NOT NULL); 2024-04-26T02:23:53.135 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.135 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.135 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.135 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.136 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.136 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.136 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.136 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.136 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.136 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.136 INFO:tasks.workunit.client.0.smithi087.stderr:SELECT LENGTH(text) FROM rand; 2024-04-26T02:23:53.136 INFO:tasks.workunit.client.0.smithi087.stderr:DROP TABLE rand;' 2024-04-26T02:23:53.136 INFO:tasks.workunit.client.0.smithi087.stderr:CREATE TABLE IF NOT EXISTS rand(text BLOB NOT NULL); 2024-04-26T02:23:53.136 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.136 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.137 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.137 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.137 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.137 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.137 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.137 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.137 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.137 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:53.137 INFO:tasks.workunit.client.0.smithi087.stderr:SELECT LENGTH(text) FROM rand; 2024-04-26T02:23:53.137 INFO:tasks.workunit.client.0.smithi087.stderr:DROP TABLE rand;+ '[' '' '!=' b ']' 2024-04-26T02:23:53.137 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 2024-04-26T02:23:53.137 INFO:tasks.workunit.client.0.smithi087.stderr:+ sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd '.open file:///cephsqlite:test_libcephsqlite.sh/baz.db?vfs=ceph' -cmd '.output stdout' 2024-04-26T02:23:54.294 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:54.295 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:54.295 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:54.295 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:54.295 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:54.295 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:54.295 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:54.295 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:54.295 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:54.295 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:54.530 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-26T02:23:54.530 INFO:tasks.workunit.client.0.smithi087.stderr:real 0m1.394s 2024-04-26T02:23:54.530 INFO:tasks.workunit.client.0.smithi087.stderr:user 0m0.154s 2024-04-26T02:23:54.530 INFO:tasks.workunit.client.0.smithi087.stderr:sys 0m0.095s 2024-04-26T02:23:54.530 INFO:tasks.workunit.client.0.smithi087.stderr:+ sqlite 2024-04-26T02:23:54.530 INFO:tasks.workunit.client.0.smithi087.stderr:++ repeat 100 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.530 INFO:tasks.workunit.client.0.smithi087.stderr:++ n=100 2024-04-26T02:23:54.530 INFO:tasks.workunit.client.0.smithi087.stderr:++ shift 2024-04-26T02:23:54.530 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i = 0 )) 2024-04-26T02:23:54.531 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.531 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.531 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.531 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.531 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.531 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.531 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.531 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.531 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.531 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.531 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.531 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.531 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.531 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.531 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.532 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.532 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.532 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.532 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.532 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.532 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.532 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.533 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.533 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.533 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.533 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.533 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.533 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.533 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.533 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.533 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.533 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.534 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.534 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.534 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.534 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.534 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.534 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.534 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.534 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.535 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.535 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.535 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.535 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.535 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.535 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.535 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.535 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.535 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.535 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.535 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.535 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.535 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.535 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.535 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.536 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.536 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.536 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.536 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.536 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.536 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.536 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.536 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.536 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.536 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.536 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.536 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.536 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.536 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.536 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.537 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.538 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.538 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.538 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.538 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.538 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.538 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.538 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.538 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.538 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.538 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.538 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.538 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.539 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.539 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.539 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.539 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.539 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.539 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.539 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.539 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.539 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.539 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.539 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.541 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.542 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.542 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.542 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.542 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.542 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.542 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.542 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.542 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.543 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.545 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.545 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.545 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.545 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.545 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.545 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.545 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.545 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.545 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.545 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.545 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.546 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.546 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.546 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.546 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.546 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.546 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.546 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.546 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.546 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.546 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.546 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.546 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.546 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.546 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.546 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.547 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.548 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.548 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.548 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.548 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.548 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.548 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.548 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.548 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.548 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.548 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.548 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.548 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.548 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.548 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.548 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.549 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.549 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.549 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.549 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.549 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.549 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.549 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.549 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.549 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.549 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.550 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.550 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.550 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.550 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.550 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.550 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.550 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.550 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.550 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.550 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.550 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.550 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.550 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.550 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.551 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.551 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.551 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.551 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.551 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.551 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.551 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.551 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.551 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.551 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.551 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.552 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.552 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.552 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.552 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.552 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.552 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.552 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.552 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.552 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.552 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.552 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.552 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.552 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.552 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.553 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.553 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.553 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.553 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.553 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.553 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.554 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.554 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.554 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.554 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.554 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.554 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.554 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.554 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.554 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.554 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo 'INSERT INTO rand (text) VALUES (RANDOMBLOB(4096));' 2024-04-26T02:23:54.554 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( ++i )) 2024-04-26T02:23:54.554 INFO:tasks.workunit.client.0.smithi087.stderr:++ (( i < 100 )) 2024-04-26T02:23:54.554 INFO:tasks.workunit.client.0.smithi087.stderr:+ background= 2024-04-26T02:23:54.554 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' '' = b ']' 2024-04-26T02:23:54.555 INFO:tasks.workunit.client.0.smithi087.stderr:++ cat 2024-04-26T02:23:54.555 INFO:tasks.workunit.client.0.smithi087.stderr:+ a='BEGIN TRANSACTION; 2024-04-26T02:23:54.555 INFO:tasks.workunit.client.0.smithi087.stderr:CREATE TABLE IF NOT EXISTS rand(text BLOB NOT NULL); 2024-04-26T02:23:54.555 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.555 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.555 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.555 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.555 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.555 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.556 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.556 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.556 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.556 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.556 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.556 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.556 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.556 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.556 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.556 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.556 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.556 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.556 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.556 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.557 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.558 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.558 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.558 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.558 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.558 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.558 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.558 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.558 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.558 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.558 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.558 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.558 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.558 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.558 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.558 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.559 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.559 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.559 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.559 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.559 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.559 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.559 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.559 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.559 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.562 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.562 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.562 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.562 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.562 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.563 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.563 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.563 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.563 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.563 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.564 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.564 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.564 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.564 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.564 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.564 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.564 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.564 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.564 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.564 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.564 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.564 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.564 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.564 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.565 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.565 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.565 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.565 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.565 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.565 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.565 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.565 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.565 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.565 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.566 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.566 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.566 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.566 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.566 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.566 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.566 INFO:tasks.workunit.client.0.smithi087.stderr:COMMIT; 2024-04-26T02:23:54.566 INFO:tasks.workunit.client.0.smithi087.stderr:SELECT LENGTH(text) FROM rand; 2024-04-26T02:23:54.566 INFO:tasks.workunit.client.0.smithi087.stderr:DROP TABLE rand;' 2024-04-26T02:23:54.566 INFO:tasks.workunit.client.0.smithi087.stderr:+ printf %s 'BEGIN TRANSACTION; 2024-04-26T02:23:54.567 INFO:tasks.workunit.client.0.smithi087.stderr:CREATE TABLE IF NOT EXISTS rand(text BLOB NOT NULL); 2024-04-26T02:23:54.567 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.567 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.567 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.567 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.567 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.567 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.567 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.567 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.567 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.567 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.567 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.567 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.567 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.567 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.568 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.568 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.568 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.568 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.568 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.568 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.568 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.568 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.568 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.568 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.568 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.568 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.569 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.569 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.569 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.569 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.569 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.569 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.569 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.569 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.569 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.569 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.569 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.569 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.569 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.569 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.570 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.570 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.570 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.570 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.570 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.570 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.570 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.570 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.570 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.570 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.570 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.570 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.570 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.570 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.570 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.572 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.572 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.572 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.572 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.572 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.572 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.572 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.572 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.572 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.573 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.573 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.573 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.573 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.573 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.573 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.573 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.573 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.573 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.573 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.574 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.574 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.574 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.574 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.574 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.574 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.574 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.574 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.574 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.574 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.574 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.574 INFO:tasks.workunit.client.0.smithi087.stderr:COMMIT; 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:SELECT LENGTH(text) FROM rand; 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:DROP TABLE rand;' 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:BEGIN TRANSACTION; 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:CREATE TABLE IF NOT EXISTS rand(text BLOB NOT NULL); 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.575 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.576 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.576 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.576 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.576 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.576 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.576 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.576 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.576 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.576 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.576 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.576 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.576 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.576 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.576 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.576 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.577 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.578 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.578 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.578 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.578 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.578 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.578 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.578 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.578 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.578 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.578 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.578 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.578 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.578 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.578 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.578 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.579 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.580 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.580 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.580 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.580 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.580 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.580 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.580 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.580 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.580 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.580 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.580 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.580 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.580 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.580 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.581 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.581 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.581 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.581 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.581 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.581 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.581 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.581 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.581 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.581 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.581 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.581 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO rand (text) VALUES (RANDOMBLOB(4096)); 2024-04-26T02:23:54.581 INFO:tasks.workunit.client.0.smithi087.stderr:COMMIT; 2024-04-26T02:23:54.581 INFO:tasks.workunit.client.0.smithi087.stderr:SELECT LENGTH(text) FROM rand; 2024-04-26T02:23:54.581 INFO:tasks.workunit.client.0.smithi087.stderr:DROP TABLE rand;+ '[' '' '!=' b ']' 2024-04-26T02:23:54.582 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 2024-04-26T02:23:54.582 INFO:tasks.workunit.client.0.smithi087.stderr:+ sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd '.open file:///cephsqlite:test_libcephsqlite.sh/baz.db?vfs=ceph' -cmd '.output stdout' 2024-04-26T02:23:55.129 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.130 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.130 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.130 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.130 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.130 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.130 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.130 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.130 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.130 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.130 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.130 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.130 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.130 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.130 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.131 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.131 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.131 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.131 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.131 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.131 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.131 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.131 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.131 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.131 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.131 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.131 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.132 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.133 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.133 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.133 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.133 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.133 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.133 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.133 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.133 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.133 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.133 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.133 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.133 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.133 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.133 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.133 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.134 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.134 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.134 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.134 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.134 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.134 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.134 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.134 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.134 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.134 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.134 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.134 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.134 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.134 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.134 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.136 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.136 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.136 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.136 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.136 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.136 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.136 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.136 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.136 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.136 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.136 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:55.136 INFO:tasks.workunit.client.0.smithi087.stdout:4096 2024-04-26T02:23:56.206 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-26T02:23:56.206 INFO:tasks.workunit.client.0.smithi087.stderr:real 0m1.684s 2024-04-26T02:23:56.206 INFO:tasks.workunit.client.0.smithi087.stderr:user 0m0.182s 2024-04-26T02:23:56.206 INFO:tasks.workunit.client.0.smithi087.stderr:sys 0m0.151s 2024-04-26T02:23:56.206 INFO:tasks.workunit.client.0.smithi087.stderr:+ striper rm baz.db 2024-04-26T02:23:56.206 INFO:tasks.workunit.client.0.smithi087.stderr:+ rados --pool=cephsqlite --namespace=test_libcephsqlite.sh --striper rm baz.db 2024-04-26T02:23:56.261 INFO:tasks.workunit.client.0.smithi087.stderr:+ date 2024-04-26T02:23:56.261 INFO:tasks.workunit.client.0.smithi087.stdout:Fri Apr 26 02:23:56 AM UTC 2024 2024-04-26T02:23:56.262 INFO:tasks.workunit.client.0.smithi087.stderr:+ sqlite b 2024-04-26T02:23:56.262 INFO:tasks.workunit.client.0.smithi087.stderr:+ background=b 2024-04-26T02:23:56.262 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' b = b ']' 2024-04-26T02:23:56.262 INFO:tasks.workunit.client.0.smithi087.stderr:+ shift 2024-04-26T02:23:56.262 INFO:tasks.workunit.client.0.smithi087.stderr:++ cat 2024-04-26T02:23:56.263 INFO:tasks.workunit.client.0.smithi087.stderr:+ a='CREATE TABLE foo (a BLOB); 2024-04-26T02:23:56.263 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO foo VALUES ("start"); 2024-04-26T02:23:56.263 INFO:tasks.workunit.client.0.smithi087.stderr:WITH RECURSIVE c(x) AS 2024-04-26T02:23:56.263 INFO:tasks.workunit.client.0.smithi087.stderr: ( 2024-04-26T02:23:56.263 INFO:tasks.workunit.client.0.smithi087.stderr: VALUES(1) 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr: UNION ALL 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr: SELECT x+1 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr: FROM c 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr: ) 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO foo (a) 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr: SELECT RANDOMBLOB(1<<20) 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr: FROM c 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr: LIMIT (1<<20);' 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr:+ printf %s 'CREATE TABLE foo (a BLOB); 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO foo VALUES ("start"); 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr:WITH RECURSIVE c(x) AS 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr: ( 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr: VALUES(1) 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr: UNION ALL 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr: SELECT x+1 2024-04-26T02:23:56.264 INFO:tasks.workunit.client.0.smithi087.stderr: FROM c 2024-04-26T02:23:56.265 INFO:tasks.workunit.client.0.smithi087.stderr: ) 2024-04-26T02:23:56.265 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO foo (a) 2024-04-26T02:23:56.265 INFO:tasks.workunit.client.0.smithi087.stderr: SELECT RANDOMBLOB(1<<20) 2024-04-26T02:23:56.265 INFO:tasks.workunit.client.0.smithi087.stderr: FROM c 2024-04-26T02:23:56.265 INFO:tasks.workunit.client.0.smithi087.stderr: LIMIT (1<<20);' 2024-04-26T02:23:56.265 INFO:tasks.workunit.client.0.smithi087.stderr:CREATE TABLE foo (a BLOB); 2024-04-26T02:23:56.265 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO foo VALUES ("start"); 2024-04-26T02:23:56.265 INFO:tasks.workunit.client.0.smithi087.stderr:WITH RECURSIVE c(x) AS 2024-04-26T02:23:56.265 INFO:tasks.workunit.client.0.smithi087.stderr: ( 2024-04-26T02:23:56.265 INFO:tasks.workunit.client.0.smithi087.stderr: VALUES(1) 2024-04-26T02:23:56.265 INFO:tasks.workunit.client.0.smithi087.stderr: UNION ALL 2024-04-26T02:23:56.265 INFO:tasks.workunit.client.0.smithi087.stderr: SELECT x+1 2024-04-26T02:23:56.265 INFO:tasks.workunit.client.0.smithi087.stderr: FROM c 2024-04-26T02:23:56.265 INFO:tasks.workunit.client.0.smithi087.stderr: ) 2024-04-26T02:23:56.265 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO foo (a) 2024-04-26T02:23:56.266 INFO:tasks.workunit.client.0.smithi087.stderr: SELECT RANDOMBLOB(1<<20) 2024-04-26T02:23:56.266 INFO:tasks.workunit.client.0.smithi087.stderr: FROM c 2024-04-26T02:23:56.266 INFO:tasks.workunit.client.0.smithi087.stderr: LIMIT (1<<20);+ '[' b '!=' b ']' 2024-04-26T02:23:56.266 INFO:tasks.workunit.client.0.smithi087.stderr:+ sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd '.open file:///cephsqlite:test_libcephsqlite.sh/baz.db?vfs=ceph' -cmd '.output stdout' 2024-04-26T02:23:56.266 INFO:tasks.workunit.client.0.smithi087.stderr:+ sleep 10 2024-04-26T02:24:06.265 INFO:tasks.workunit.client.0.smithi087.stdout:done 2024-04-26T02:24:06.265 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo done 2024-04-26T02:24:06.266 INFO:tasks.workunit.client.0.smithi087.stderr:+ jobs -l 2024-04-26T02:24:06.266 INFO:tasks.workunit.client.0.smithi087.stdout:[1]+ 30124 Running sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd ".open file:///$pool:$ns/baz.db?vfs=ceph" -cmd '.output stdout' <<< "$a" & 2024-04-26T02:24:06.266 INFO:tasks.workunit.client.0.smithi087.stderr:++ jobs -p 2024-04-26T02:24:06.267 INFO:tasks.workunit.client.0.smithi087.stderr:+ kill -KILL -- 30124 2024-04-26T02:24:06.267 INFO:tasks.workunit.client.0.smithi087.stderr:+ date 2024-04-26T02:24:06.267 INFO:tasks.workunit.client.0.smithi087.stdout:Fri Apr 26 02:24:06 AM UTC 2024 2024-04-26T02:24:06.268 INFO:tasks.workunit.client.0.smithi087.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_libcephsqlite.sh: line 94: 30124 Killed sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd ".open file:///$pool:$ns/baz.db?vfs=ceph" -cmd '.output stdout' <<< "$a" 2024-04-26T02:24:06.268 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 2024-04-26T02:24:06.269 INFO:tasks.workunit.client.0.smithi087.stderr:+ date 2024-04-26T02:24:06.269 INFO:tasks.workunit.client.0.smithi087.stdout:Fri Apr 26 02:24:06 AM UTC 2024 2024-04-26T02:24:06.270 INFO:tasks.workunit.client.0.smithi087.stderr:++ sqlite 2024-04-26T02:24:06.270 INFO:tasks.workunit.client.0.smithi087.stderr:++ background= 2024-04-26T02:24:06.271 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' = b ']' 2024-04-26T02:24:06.272 INFO:tasks.workunit.client.0.smithi087.stderr:+++ cat 2024-04-26T02:24:06.274 INFO:tasks.workunit.client.0.smithi087.stderr:++ a='SELECT COUNT(*) FROM foo;' 2024-04-26T02:24:06.275 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %s 'SELECT COUNT(*) FROM foo;' 2024-04-26T02:24:06.275 INFO:tasks.workunit.client.0.smithi087.stderr:SELECT COUNT(*) FROM foo;++ '[' '' '!=' b ']' 2024-04-26T02:24:06.275 INFO:tasks.workunit.client.0.smithi087.stderr:++ wait 2024-04-26T02:24:06.275 INFO:tasks.workunit.client.0.smithi087.stderr:++ sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd '.open file:///cephsqlite:test_libcephsqlite.sh/baz.db?vfs=ceph' -cmd '.output stdout' 2024-04-26T02:24:34.792 INFO:tasks.workunit.client.0.smithi087.stderr:+ n=1 2024-04-26T02:24:34.793 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T02:24:34.793 INFO:tasks.workunit.client.0.smithi087.stderr:+ striper rm baz.db 2024-04-26T02:24:34.793 INFO:tasks.workunit.client.0.smithi087.stderr:+ rados --pool=cephsqlite --namespace=test_libcephsqlite.sh --striper rm baz.db 2024-04-26T02:24:34.869 INFO:tasks.workunit.client.0.smithi087.stderr:+ date 2024-04-26T02:24:34.870 INFO:tasks.workunit.client.0.smithi087.stdout:Fri Apr 26 02:24:34 AM UTC 2024 2024-04-26T02:24:34.870 INFO:tasks.workunit.client.0.smithi087.stderr:+ sqlite b 2024-04-26T02:24:34.871 INFO:tasks.workunit.client.0.smithi087.stderr:+ background=b 2024-04-26T02:24:34.871 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' b = b ']' 2024-04-26T02:24:34.871 INFO:tasks.workunit.client.0.smithi087.stderr:+ shift 2024-04-26T02:24:34.871 INFO:tasks.workunit.client.0.smithi087.stderr:++ cat 2024-04-26T02:24:34.871 INFO:tasks.workunit.client.0.smithi087.stderr:+ a='CREATE TABLE foo (a BLOB); 2024-04-26T02:24:34.872 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO foo VALUES ("start"); 2024-04-26T02:24:34.872 INFO:tasks.workunit.client.0.smithi087.stderr:WITH RECURSIVE c(x) AS 2024-04-26T02:24:34.872 INFO:tasks.workunit.client.0.smithi087.stderr: ( 2024-04-26T02:24:34.872 INFO:tasks.workunit.client.0.smithi087.stderr: VALUES(1) 2024-04-26T02:24:34.872 INFO:tasks.workunit.client.0.smithi087.stderr: UNION ALL 2024-04-26T02:24:34.872 INFO:tasks.workunit.client.0.smithi087.stderr: SELECT x+1 2024-04-26T02:24:34.872 INFO:tasks.workunit.client.0.smithi087.stderr: FROM c 2024-04-26T02:24:34.873 INFO:tasks.workunit.client.0.smithi087.stderr: ) 2024-04-26T02:24:34.873 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO foo (a) 2024-04-26T02:24:34.873 INFO:tasks.workunit.client.0.smithi087.stderr: SELECT RANDOMBLOB(1<<20) 2024-04-26T02:24:34.873 INFO:tasks.workunit.client.0.smithi087.stderr: FROM c 2024-04-26T02:24:34.873 INFO:tasks.workunit.client.0.smithi087.stderr: LIMIT (1<<20);' 2024-04-26T02:24:34.873 INFO:tasks.workunit.client.0.smithi087.stderr:+ printf %s 'CREATE TABLE foo (a BLOB); 2024-04-26T02:24:34.873 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO foo VALUES ("start"); 2024-04-26T02:24:34.873 INFO:tasks.workunit.client.0.smithi087.stderr:WITH RECURSIVE c(x) AS 2024-04-26T02:24:34.873 INFO:tasks.workunit.client.0.smithi087.stderr: ( 2024-04-26T02:24:34.873 INFO:tasks.workunit.client.0.smithi087.stderr: VALUES(1) 2024-04-26T02:24:34.873 INFO:tasks.workunit.client.0.smithi087.stderr: UNION ALL 2024-04-26T02:24:34.873 INFO:tasks.workunit.client.0.smithi087.stderr: SELECT x+1 2024-04-26T02:24:34.873 INFO:tasks.workunit.client.0.smithi087.stderr: FROM c 2024-04-26T02:24:34.873 INFO:tasks.workunit.client.0.smithi087.stderr: ) 2024-04-26T02:24:34.873 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO foo (a) 2024-04-26T02:24:34.874 INFO:tasks.workunit.client.0.smithi087.stderr: SELECT RANDOMBLOB(1<<20) 2024-04-26T02:24:34.874 INFO:tasks.workunit.client.0.smithi087.stderr: FROM c 2024-04-26T02:24:34.874 INFO:tasks.workunit.client.0.smithi087.stderr: LIMIT (1<<20);' 2024-04-26T02:24:34.874 INFO:tasks.workunit.client.0.smithi087.stderr:CREATE TABLE foo (a BLOB); 2024-04-26T02:24:34.874 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO foo VALUES ("start"); 2024-04-26T02:24:34.874 INFO:tasks.workunit.client.0.smithi087.stderr:WITH RECURSIVE c(x) AS 2024-04-26T02:24:34.874 INFO:tasks.workunit.client.0.smithi087.stderr: ( 2024-04-26T02:24:34.874 INFO:tasks.workunit.client.0.smithi087.stderr: VALUES(1) 2024-04-26T02:24:34.874 INFO:tasks.workunit.client.0.smithi087.stderr: UNION ALL 2024-04-26T02:24:34.874 INFO:tasks.workunit.client.0.smithi087.stderr: SELECT x+1 2024-04-26T02:24:34.874 INFO:tasks.workunit.client.0.smithi087.stderr: FROM c 2024-04-26T02:24:34.874 INFO:tasks.workunit.client.0.smithi087.stderr: ) 2024-04-26T02:24:34.874 INFO:tasks.workunit.client.0.smithi087.stderr:INSERT INTO foo (a) 2024-04-26T02:24:34.874 INFO:tasks.workunit.client.0.smithi087.stderr: SELECT RANDOMBLOB(1<<20) 2024-04-26T02:24:34.874 INFO:tasks.workunit.client.0.smithi087.stderr: FROM c 2024-04-26T02:24:34.875 INFO:tasks.workunit.client.0.smithi087.stderr: LIMIT (1<<20);+ '[' b '!=' b ']' 2024-04-26T02:24:34.875 INFO:tasks.workunit.client.0.smithi087.stderr:+ sleep 20 2024-04-26T02:24:34.875 INFO:tasks.workunit.client.0.smithi087.stderr:+ sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd '.open file:///cephsqlite:test_libcephsqlite.sh/baz.db?vfs=ceph' -cmd '.output stdout' 2024-04-26T02:24:54.873 INFO:tasks.workunit.client.0.smithi087.stderr:+ jobs -l 2024-04-26T02:24:54.874 INFO:tasks.workunit.client.0.smithi087.stdout:[1]+ 30192 Running sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd ".open file:///$pool:$ns/baz.db?vfs=ceph" -cmd '.output stdout' <<< "$a" & 2024-04-26T02:24:54.874 INFO:tasks.workunit.client.0.smithi087.stderr:++ jobs -p 2024-04-26T02:24:54.874 INFO:tasks.workunit.client.0.smithi087.stderr:+ kill -STOP -- 30192 2024-04-26T02:24:54.875 INFO:tasks.workunit.client.0.smithi087.stderr:+ sleep 45 2024-04-26T02:25:39.876 INFO:tasks.workunit.client.0.smithi087.stderr:+ date 2024-04-26T02:25:39.877 INFO:tasks.workunit.client.0.smithi087.stdout:Fri Apr 26 02:25:39 AM UTC 2024 2024-04-26T02:25:39.878 INFO:tasks.workunit.client.0.smithi087.stderr:++ jobs -p 2024-04-26T02:25:39.878 INFO:tasks.workunit.client.0.smithi087.stderr:+ kill -CONT -- 30192 2024-04-26T02:25:39.878 INFO:tasks.workunit.client.0.smithi087.stderr:+ sleep 10 2024-04-26T02:25:39.912 INFO:tasks.workunit.client.0.smithi087.stderr:Error: near line 3: stepping, disk I/O error (10) 2024-04-26T02:25:49.880 INFO:tasks.workunit.client.0.smithi087.stderr:+ date 2024-04-26T02:25:49.881 INFO:tasks.workunit.client.0.smithi087.stdout:Fri Apr 26 02:25:49 AM UTC 2024 2024-04-26T02:25:49.881 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 2024-04-26T02:25:49.881 INFO:tasks.workunit.client.0.smithi087.stderr:+ date 2024-04-26T02:25:49.882 INFO:tasks.workunit.client.0.smithi087.stdout:Fri Apr 26 02:25:49 AM UTC 2024 2024-04-26T02:25:49.883 INFO:tasks.workunit.client.0.smithi087.stderr:++ sqlite 2024-04-26T02:25:49.884 INFO:tasks.workunit.client.0.smithi087.stderr:++ background= 2024-04-26T02:25:49.884 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' = b ']' 2024-04-26T02:25:49.884 INFO:tasks.workunit.client.0.smithi087.stderr:+++ cat 2024-04-26T02:25:49.885 INFO:tasks.workunit.client.0.smithi087.stderr:++ a='SELECT COUNT(*) FROM foo;' 2024-04-26T02:25:49.885 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %s 'SELECT COUNT(*) FROM foo;' 2024-04-26T02:25:49.886 INFO:tasks.workunit.client.0.smithi087.stderr:SELECT COUNT(*) FROM foo;++ '[' '' '!=' b ']' 2024-04-26T02:25:49.886 INFO:tasks.workunit.client.0.smithi087.stderr:++ wait 2024-04-26T02:25:49.886 INFO:tasks.workunit.client.0.smithi087.stderr:++ sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd '.open file:///cephsqlite:test_libcephsqlite.sh/baz.db?vfs=ceph' -cmd '.output stdout' 2024-04-26T02:25:50.738 INFO:tasks.workunit.client.0.smithi087.stderr:+ n=1 2024-04-26T02:25:50.739 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T02:25:50.739 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-26T02:25:50.739 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-26T02:25:50.793 INFO:tasks.workunit:Stopping ['rados/test_libcephsqlite.sh cephsqlite'] on client.0... 2024-04-26T02:25:50.794 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-26T02:25:51.141 DEBUG:teuthology.parallel:result is None 2024-04-26T02:25:51.141 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-26T02:25:51.153 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-26T02:25:51.154 DEBUG:teuthology.orchestra.run.smithi087:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-26T02:25:51.211 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-26T02:25:51.212 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-26T02:25:51.222 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T02:25:51.222 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:51.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc4784b4640 1 Processor -- start 2024-04-26T02:25:51.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc4784b4640 1 -- start start 2024-04-26T02:25:51.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc4784b4640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4700ad2c0 con 0x7fc4700a84d0 2024-04-26T02:25:51.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc4784b4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc470142a40 con 0x7fc470143290 2024-04-26T02:25:51.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc4784b4640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4700abfc0 con 0x7fc4700aaa50 2024-04-26T02:25:51.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc476250640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fc4700aaa50 0x7fc4700aae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38820/0 (socket says 172.21.15.87:38820) 2024-04-26T02:25:51.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc476250640 1 -- 172.21.15.87:0/1104574118 learned_addr learned my addr 172.21.15.87:0/1104574118 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:51.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc476a51640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fc4700a84d0 0x7fc4700a8980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51726/0 (socket says 172.21.15.87:51726) 2024-04-26T02:25:51.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc47524e640 1 -- 172.21.15.87:0/1104574118 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2508529028 0 0) 0x7fc4680014a0 con 0x7fc4700a84d0 2024-04-26T02:25:51.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc47524e640 1 -- 172.21.15.87:0/1104574118 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc45c003530 con 0x7fc4700a84d0 2024-04-26T02:25:51.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc47524e640 1 -- 172.21.15.87:0/1104574118 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1520658237 0 0) 0x7fc4700abfc0 con 0x7fc4700aaa50 2024-04-26T02:25:51.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc47524e640 1 -- 172.21.15.87:0/1104574118 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4680014a0 con 0x7fc4700aaa50 2024-04-26T02:25:51.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc47524e640 1 -- 172.21.15.87:0/1104574118 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1640739434 0 0) 0x7fc45c003530 con 0x7fc4700a84d0 2024-04-26T02:25:51.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc47524e640 1 -- 172.21.15.87:0/1104574118 >> v1:172.21.15.87:6790/0 conn(0x7fc4700aaa50 legacy=0x7fc4700aae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:51.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc47524e640 1 -- 172.21.15.87:0/1104574118 >> v1:172.21.15.122:6789/0 conn(0x7fc470143290 legacy=0x7fc470138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:51.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc47524e640 1 -- 172.21.15.87:0/1104574118 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4700ac190 con 0x7fc4700a84d0 2024-04-26T02:25:51.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc47524e640 1 -- 172.21.15.87:0/1104574118 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc468001b70 con 0x7fc4700a84d0 2024-04-26T02:25:51.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc47524e640 1 -- 172.21.15.87:0/1104574118 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc468001dd0 con 0x7fc4700a84d0 2024-04-26T02:25:51.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc47524e640 1 -- 172.21.15.87:0/1104574118 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc468004b70 con 0x7fc4700a84d0 2024-04-26T02:25:51.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc4784b4640 1 -- 172.21.15.87:0/1104574118 >> v1:172.21.15.87:6789/0 conn(0x7fc4700a84d0 legacy=0x7fc4700a8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:51.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc4784b4640 1 -- 172.21.15.87:0/1104574118 shutdown_connections 2024-04-26T02:25:51.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc4784b4640 1 -- 172.21.15.87:0/1104574118 >> 172.21.15.87:0/1104574118 conn(0x7fc4700b8180 msgr2=0x7fc4700b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:51.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc4784b4640 1 -- 172.21.15.87:0/1104574118 shutdown_connections 2024-04-26T02:25:51.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc4784b4640 1 -- 172.21.15.87:0/1104574118 wait complete. 2024-04-26T02:25:51.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc4784b4640 1 Processor -- start 2024-04-26T02:25:51.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.376+0000 7fc4784b4640 1 -- start start 2024-04-26T02:25:51.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc4784b4640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4701db280 con 0x7fc470143290 2024-04-26T02:25:51.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc4784b4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4701419c0 con 0x7fc4700a84d0 2024-04-26T02:25:51.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc4784b4640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc470141320 con 0x7fc4700aaa50 2024-04-26T02:25:51.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc475a4f640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fc4700aaa50 0x7fc4701dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38830/0 (socket says 172.21.15.87:38830) 2024-04-26T02:25:51.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc475a4f640 1 -- 172.21.15.87:0/4264053879 learned_addr learned my addr 172.21.15.87:0/4264053879 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:51.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc476a51640 1 --1- 172.21.15.87:0/4264053879 >> v1:172.21.15.87:6789/0 conn(0x7fc470143290 0x7fc470140c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51734/0 (socket says 172.21.15.87:51734) 2024-04-26T02:25:51.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3549897198 0 0) 0x7fc470141320 con 0x7fc4700aaa50 2024-04-26T02:25:51.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc454003530 con 0x7fc4700aaa50 2024-04-26T02:25:51.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2891619872 0 0) 0x7fc4701db280 con 0x7fc470143290 2024-04-26T02:25:51.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc470141320 con 0x7fc470143290 2024-04-26T02:25:51.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3462990314 0 0) 0x7fc454003530 con 0x7fc4700aaa50 2024-04-26T02:25:51.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc4701db280 con 0x7fc4700aaa50 2024-04-26T02:25:51.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc4600018b0 con 0x7fc4700aaa50 2024-04-26T02:25:51.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1451799613 0 0) 0x7fc470141320 con 0x7fc470143290 2024-04-26T02:25:51.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc454003530 con 0x7fc470143290 2024-04-26T02:25:51.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc468001260 con 0x7fc470143290 2024-04-26T02:25:51.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3702968296 0 0) 0x7fc4701419c0 con 0x7fc4700a84d0 2024-04-26T02:25:51.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc470141320 con 0x7fc4700a84d0 2024-04-26T02:25:51.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2307361539 0 0) 0x7fc4701db280 con 0x7fc4700aaa50 2024-04-26T02:25:51.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 >> v1:172.21.15.122:6789/0 conn(0x7fc4700a84d0 legacy=0x7fc470137090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:51.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 >> v1:172.21.15.87:6789/0 conn(0x7fc470143290 legacy=0x7fc470140c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:51.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4701414f0 con 0x7fc4700aaa50 2024-04-26T02:25:51.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc4784b4640 1 -- 172.21.15.87:0/4264053879 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc470136160 con 0x7fc4700aaa50 2024-04-26T02:25:51.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc4784b4640 1 -- 172.21.15.87:0/4264053879 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc470138d30 con 0x7fc4700aaa50 2024-04-26T02:25:51.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc4784b4640 1 -- 172.21.15.87:0/4264053879 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc47004fd20 con 0x7fc4700aaa50 2024-04-26T02:25:51.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc4600036a0 con 0x7fc4700aaa50 2024-04-26T02:25:51.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fc460003a70 con 0x7fc4700aaa50 2024-04-26T02:25:51.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7fc460003c90 con 0x7fc4700aaa50 2024-04-26T02:25:51.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.380+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7fc4600812b0 con 0x7fc4700aaa50 2024-04-26T02:25:51.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.384+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fc460052120 con 0x7fc4700aaa50 2024-04-26T02:25:51.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.544+0000 7fc4784b4640 1 -- 172.21.15.87:0/4264053879 --> v1:172.21.15.122:6816/416193625 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc470136c90 con 0x7fc4540683d0 2024-04-26T02:25:51.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.552+0000 7fc466ffd640 1 -- 172.21.15.87:0/4264053879 <== mgr.4108 v1:172.21.15.122:6816/416193625 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+117694 (unknown 2965378022 0 2277588350) 0x7fc470136c90 con 0x7fc4540683d0 2024-04-26T02:25:51.558 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:25:51.559 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-26T02:25:51.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.556+0000 7fc4784b4640 1 -- 172.21.15.87:0/4264053879 >> v1:172.21.15.122:6816/416193625 conn(0x7fc4540683d0 legacy=0x7fc45406a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:51.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.556+0000 7fc4784b4640 1 -- 172.21.15.87:0/4264053879 >> v1:172.21.15.87:6790/0 conn(0x7fc4700aaa50 legacy=0x7fc4701dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:51.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.556+0000 7fc4784b4640 1 -- 172.21.15.87:0/4264053879 shutdown_connections 2024-04-26T02:25:51.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.556+0000 7fc4784b4640 1 -- 172.21.15.87:0/4264053879 >> 172.21.15.87:0/4264053879 conn(0x7fc4700b8180 msgr2=0x7fc4700b7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:51.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.556+0000 7fc4784b4640 1 -- 172.21.15.87:0/4264053879 shutdown_connections 2024-04-26T02:25:51.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.556+0000 7fc4784b4640 1 -- 172.21.15.87:0/4264053879 wait complete. 2024-04-26T02:25:51.575 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-04-26T02:25:51.415976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3568989480,"num_objects":880,"num_object_clones":0,"num_object_copies":1760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":2609,"num_read_kb":2604,"num_write":159870,"num_write_kb":5638989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161378,"ondisk_log_size":161378,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":9156376,"kb_used_data":6640344,"kb_used_omap":12,"kb_used_meta":2515955,"kb_avail":745818344,"statfs":{"total":773094113280,"available":763717984256,"internally_reserved":0,"allocated":6799712256,"data_stored":6837093920,"data_compressed":194304976,"data_compressed_allocated":202137600,"data_compressed_original":244916224,"omap_allocated":12751,"internal_metadata":2576338481},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36864,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003159"},"pg_stats":[{"pgid":"3.1f","version":"22'2749","reported_seq":2768,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.658393+0000","last_change":"2024-04-26T02:21:37.492661+0000","last_active":"2024-04-26T02:24:34.658393+0000","last_peered":"2024-04-26T02:24:34.658393+0000","last_clean":"2024-04-26T02:24:34.658393+0000","last_became_active":"2024-04-26T02:21:37.491078+0000","last_became_peered":"2024-04-26T02:21:37.491078+0000","last_unstale":"2024-04-26T02:24:34.658393+0000","last_undegraded":"2024-04-26T02:24:34.658393+0000","last_fullsized":"2024-04-26T02:24:34.658393+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":2749,"log_dups_size":0,"ondisk_log_size":2749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:56:51.567611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":71,"num_write":2855,"num_write_kb":123586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"23'6576","reported_seq":6592,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:42.266530+0000","last_change":"2024-04-26T02:21:37.485093+0000","last_active":"2024-04-26T02:24:42.266530+0000","last_peered":"2024-04-26T02:24:42.266530+0000","last_clean":"2024-04-26T02:24:42.266530+0000","last_became_active":"2024-04-26T02:21:37.484287+0000","last_became_peered":"2024-04-26T02:21:37.484287+0000","last_unstale":"2024-04-26T02:24:42.266530+0000","last_undegraded":"2024-04-26T02:24:42.266530+0000","last_fullsized":"2024-04-26T02:24:42.266530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":6576,"log_dups_size":0,"ondisk_log_size":6576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:56:08.755136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6549,"num_write_kb":206212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"23'5724","reported_seq":5759,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:47.550994+0000","last_change":"2024-04-26T02:21:37.498519+0000","last_active":"2024-04-26T02:24:47.550994+0000","last_peered":"2024-04-26T02:24:47.550994+0000","last_clean":"2024-04-26T02:24:47.550994+0000","last_became_active":"2024-04-26T02:21:37.497700+0000","last_became_peered":"2024-04-26T02:21:37.497700+0000","last_unstale":"2024-04-26T02:24:47.550994+0000","last_undegraded":"2024-04-26T02:24:47.550994+0000","last_fullsized":"2024-04-26T02:24:47.550994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":5724,"log_dups_size":0,"ondisk_log_size":5724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:02:58.912915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":76,"num_read_kb":124,"num_write":5736,"num_write_kb":155565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"22'4059","reported_seq":4077,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.734233+0000","last_change":"2024-04-26T02:21:37.497650+0000","last_active":"2024-04-26T02:24:34.734233+0000","last_peered":"2024-04-26T02:24:34.734233+0000","last_clean":"2024-04-26T02:24:34.734233+0000","last_became_active":"2024-04-26T02:21:37.496863+0000","last_became_peered":"2024-04-26T02:21:37.496863+0000","last_unstale":"2024-04-26T02:24:34.734233+0000","last_undegraded":"2024-04-26T02:24:34.734233+0000","last_fullsized":"2024-04-26T02:24:34.734233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4059,"log_dups_size":0,"ondisk_log_size":4059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:57:07.038484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":4059,"num_write_kb":185415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"23'3948","reported_seq":3972,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.141201+0000","last_change":"2024-04-26T02:21:37.491890+0000","last_active":"2024-04-26T02:24:35.141201+0000","last_peered":"2024-04-26T02:24:35.141201+0000","last_clean":"2024-04-26T02:24:35.141201+0000","last_became_active":"2024-04-26T02:21:37.490474+0000","last_became_peered":"2024-04-26T02:21:37.490474+0000","last_unstale":"2024-04-26T02:24:35.141201+0000","last_undegraded":"2024-04-26T02:24:35.141201+0000","last_fullsized":"2024-04-26T02:24:35.141201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3948,"log_dups_size":0,"ondisk_log_size":3948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:56.993069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120324096,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":34,"num_read_kb":151,"num_write":3926,"num_write_kb":190983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"23'6081","reported_seq":6102,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:41.034089+0000","last_change":"2024-04-26T02:22:35.114133+0000","last_active":"2024-04-26T02:24:41.034089+0000","last_peered":"2024-04-26T02:24:41.034089+0000","last_clean":"2024-04-26T02:24:41.034089+0000","last_became_active":"2024-04-26T02:21:37.485110+0000","last_became_peered":"2024-04-26T02:21:37.485110+0000","last_unstale":"2024-04-26T02:24:41.034089+0000","last_undegraded":"2024-04-26T02:24:41.034089+0000","last_fullsized":"2024-04-26T02:24:41.034089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":6081,"log_dups_size":0,"ondisk_log_size":6081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:39:18.389527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":6064,"num_write_kb":187005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"23'5039","reported_seq":5073,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:52.103217+0000","last_change":"2024-04-26T02:21:37.489148+0000","last_active":"2024-04-26T02:24:52.103217+0000","last_peered":"2024-04-26T02:24:52.103217+0000","last_clean":"2024-04-26T02:24:52.103217+0000","last_became_active":"2024-04-26T02:21:37.488519+0000","last_became_peered":"2024-04-26T02:21:37.488519+0000","last_unstale":"2024-04-26T02:24:52.103217+0000","last_undegraded":"2024-04-26T02:24:52.103217+0000","last_fullsized":"2024-04-26T02:24:52.103217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:41.178413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117441024,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":5034,"num_write_kb":181716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"23'7153","reported_seq":7190,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:52.126229+0000","last_change":"2024-04-26T02:21:37.496383+0000","last_active":"2024-04-26T02:24:52.126229+0000","last_peered":"2024-04-26T02:24:52.126229+0000","last_clean":"2024-04-26T02:24:52.126229+0000","last_became_active":"2024-04-26T02:21:37.495694+0000","last_became_peered":"2024-04-26T02:21:37.495694+0000","last_unstale":"2024-04-26T02:24:52.126229+0000","last_undegraded":"2024-04-26T02:24:52.126229+0000","last_fullsized":"2024-04-26T02:24:52.126229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":7153,"log_dups_size":0,"ondisk_log_size":7153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:27:30.478083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023936,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":80,"num_read_kb":65,"num_write":7184,"num_write_kb":197893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"23'8281","reported_seq":8304,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:25:39.938993+0000","last_change":"2024-04-26T02:21:37.490841+0000","last_active":"2024-04-26T02:25:39.938993+0000","last_peered":"2024-04-26T02:25:39.938993+0000","last_clean":"2024-04-26T02:25:39.938993+0000","last_became_active":"2024-04-26T02:21:37.490119+0000","last_became_peered":"2024-04-26T02:21:37.490119+0000","last_unstale":"2024-04-26T02:25:39.938993+0000","last_undegraded":"2024-04-26T02:25:39.938993+0000","last_fullsized":"2024-04-26T02:25:39.938993+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":8281,"log_dups_size":0,"ondisk_log_size":8281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:36:46.649805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137060352,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":8291,"num_write_kb":229215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"23'3756","reported_seq":3771,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.020013+0000","last_change":"2024-04-26T02:21:37.507074+0000","last_active":"2024-04-26T02:24:35.020013+0000","last_peered":"2024-04-26T02:24:35.020013+0000","last_clean":"2024-04-26T02:24:35.020013+0000","last_became_active":"2024-04-26T02:21:37.506309+0000","last_became_peered":"2024-04-26T02:21:37.506309+0000","last_unstale":"2024-04-26T02:24:35.020013+0000","last_undegraded":"2024-04-26T02:24:35.020013+0000","last_fullsized":"2024-04-26T02:24:35.020013+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3756,"log_dups_size":0,"ondisk_log_size":3756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:49:23.247821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3724,"num_write_kb":119684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"23'7177","reported_seq":7192,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:50.387812+0000","last_change":"2024-04-26T02:21:37.487697+0000","last_active":"2024-04-26T02:24:50.387812+0000","last_peered":"2024-04-26T02:24:50.387812+0000","last_clean":"2024-04-26T02:24:50.387812+0000","last_became_active":"2024-04-26T02:21:37.486868+0000","last_became_peered":"2024-04-26T02:21:37.486868+0000","last_unstale":"2024-04-26T02:24:50.387812+0000","last_undegraded":"2024-04-26T02:24:50.387812+0000","last_fullsized":"2024-04-26T02:24:50.387812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":7177,"log_dups_size":0,"ondisk_log_size":7177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:26:13.460721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7176,"num_write_kb":198373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"23'4534","reported_seq":4549,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:44.307401+0000","last_change":"2024-04-26T02:21:37.488921+0000","last_active":"2024-04-26T02:24:44.307401+0000","last_peered":"2024-04-26T02:24:44.307401+0000","last_clean":"2024-04-26T02:24:44.307401+0000","last_became_active":"2024-04-26T02:21:37.487980+0000","last_became_peered":"2024-04-26T02:21:37.487980+0000","last_unstale":"2024-04-26T02:24:44.307401+0000","last_undegraded":"2024-04-26T02:24:44.307401+0000","last_fullsized":"2024-04-26T02:24:44.307401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4534,"log_dups_size":0,"ondisk_log_size":4534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:10.065671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4488,"num_write_kb":107395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.730342+0000","last_change":"2024-04-26T02:21:31.429340+0000","last_active":"2024-04-26T02:24:34.730342+0000","last_peered":"2024-04-26T02:24:34.730342+0000","last_clean":"2024-04-26T02:24:34.730342+0000","last_became_active":"2024-04-26T02:21:31.428657+0000","last_became_peered":"2024-04-26T02:21:31.428657+0000","last_unstale":"2024-04-26T02:24:34.730342+0000","last_undegraded":"2024-04-26T02:24:34.730342+0000","last_fullsized":"2024-04-26T02:24:34.730342+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:07:12.674824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'4886","reported_seq":4909,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:48.178325+0000","last_change":"2024-04-26T02:21:37.482817+0000","last_active":"2024-04-26T02:24:48.178325+0000","last_peered":"2024-04-26T02:24:48.178325+0000","last_clean":"2024-04-26T02:24:48.178325+0000","last_became_active":"2024-04-26T02:21:37.481620+0000","last_became_peered":"2024-04-26T02:21:37.481620+0000","last_unstale":"2024-04-26T02:24:48.178325+0000","last_undegraded":"2024-04-26T02:24:48.178325+0000","last_fullsized":"2024-04-26T02:24:48.178325+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4886,"log_dups_size":0,"ondisk_log_size":4886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:26.760929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96600064,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":4875,"num_write_kb":131621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.658277+0000","last_change":"2024-04-26T02:21:31.432330+0000","last_active":"2024-04-26T02:24:34.658277+0000","last_peered":"2024-04-26T02:24:34.658277+0000","last_clean":"2024-04-26T02:24:34.658277+0000","last_became_active":"2024-04-26T02:21:31.431958+0000","last_became_peered":"2024-04-26T02:21:31.431958+0000","last_unstale":"2024-04-26T02:24:34.658277+0000","last_undegraded":"2024-04-26T02:24:34.658277+0000","last_fullsized":"2024-04-26T02:24:34.658277+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:59:02.906743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'4222","reported_seq":4237,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.143067+0000","last_change":"2024-04-26T02:21:37.488836+0000","last_active":"2024-04-26T02:24:35.143067+0000","last_peered":"2024-04-26T02:24:35.143067+0000","last_clean":"2024-04-26T02:24:35.143067+0000","last_became_active":"2024-04-26T02:21:37.487980+0000","last_became_peered":"2024-04-26T02:21:37.487980+0000","last_unstale":"2024-04-26T02:24:35.143067+0000","last_undegraded":"2024-04-26T02:24:35.143067+0000","last_fullsized":"2024-04-26T02:24:35.143067+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:07:49.262710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4175,"num_write_kb":201729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.684259+0000","last_change":"2024-04-26T02:21:31.430988+0000","last_active":"2024-04-26T02:24:34.684259+0000","last_peered":"2024-04-26T02:24:34.684259+0000","last_clean":"2024-04-26T02:24:34.684259+0000","last_became_active":"2024-04-26T02:21:31.430227+0000","last_became_peered":"2024-04-26T02:21:31.430227+0000","last_unstale":"2024-04-26T02:24:34.684259+0000","last_undegraded":"2024-04-26T02:24:34.684259+0000","last_fullsized":"2024-04-26T02:24:34.684259+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:32:21.265195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'6121","reported_seq":6142,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:54.144536+0000","last_change":"2024-04-26T02:21:37.483474+0000","last_active":"2024-04-26T02:24:54.144536+0000","last_peered":"2024-04-26T02:24:54.144536+0000","last_clean":"2024-04-26T02:24:54.144536+0000","last_became_active":"2024-04-26T02:21:37.482894+0000","last_became_peered":"2024-04-26T02:21:37.482894+0000","last_unstale":"2024-04-26T02:24:54.144536+0000","last_undegraded":"2024-04-26T02:24:54.144536+0000","last_fullsized":"2024-04-26T02:24:54.144536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":6121,"log_dups_size":0,"ondisk_log_size":6121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:14.817911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99354976,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":6134,"num_write_kb":151141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.659060+0000","last_change":"2024-04-26T02:21:31.431358+0000","last_active":"2024-04-26T02:24:34.659060+0000","last_peered":"2024-04-26T02:24:34.659060+0000","last_clean":"2024-04-26T02:24:34.659060+0000","last_became_active":"2024-04-26T02:21:31.431071+0000","last_became_peered":"2024-04-26T02:21:31.431071+0000","last_unstale":"2024-04-26T02:24:34.659060+0000","last_undegraded":"2024-04-26T02:24:34.659060+0000","last_fullsized":"2024-04-26T02:24:34.659060+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:41:07.918405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"22'4180","reported_seq":4225,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.735876+0000","last_change":"2024-04-26T02:21:37.490970+0000","last_active":"2024-04-26T02:24:34.735876+0000","last_peered":"2024-04-26T02:24:34.735876+0000","last_clean":"2024-04-26T02:24:34.735876+0000","last_became_active":"2024-04-26T02:21:37.490105+0000","last_became_peered":"2024-04-26T02:21:37.490105+0000","last_unstale":"2024-04-26T02:24:34.735876+0000","last_undegraded":"2024-04-26T02:24:34.735876+0000","last_fullsized":"2024-04-26T02:24:34.735876+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4180,"log_dups_size":0,"ondisk_log_size":4180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:17.335251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121896960,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":122,"num_read_kb":98,"num_write":4227,"num_write_kb":167240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"22'2962","reported_seq":2980,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.670117+0000","last_change":"2024-04-26T02:21:37.494426+0000","last_active":"2024-04-26T02:24:34.670117+0000","last_peered":"2024-04-26T02:24:34.670117+0000","last_clean":"2024-04-26T02:24:34.670117+0000","last_became_active":"2024-04-26T02:21:37.491249+0000","last_became_peered":"2024-04-26T02:21:37.491249+0000","last_unstale":"2024-04-26T02:24:34.670117+0000","last_undegraded":"2024-04-26T02:24:34.670117+0000","last_fullsized":"2024-04-26T02:24:34.670117+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:22:39.029300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74842112,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":7,"num_write":2966,"num_write_kb":163586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.734132+0000","last_change":"2024-04-26T02:21:32.425721+0000","last_active":"2024-04-26T02:24:34.734132+0000","last_peered":"2024-04-26T02:24:34.734132+0000","last_clean":"2024-04-26T02:24:34.734132+0000","last_became_active":"2024-04-26T02:21:32.425189+0000","last_became_peered":"2024-04-26T02:21:32.425189+0000","last_unstale":"2024-04-26T02:24:34.734132+0000","last_undegraded":"2024-04-26T02:24:34.734132+0000","last_fullsized":"2024-04-26T02:24:34.734132+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:56:47.156837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"23'3481","reported_seq":3503,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.146368+0000","last_change":"2024-04-26T02:21:37.486201+0000","last_active":"2024-04-26T02:24:35.146368+0000","last_peered":"2024-04-26T02:24:35.146368+0000","last_clean":"2024-04-26T02:24:35.146368+0000","last_became_active":"2024-04-26T02:21:37.485432+0000","last_became_peered":"2024-04-26T02:21:37.485432+0000","last_unstale":"2024-04-26T02:24:35.146368+0000","last_undegraded":"2024-04-26T02:24:35.146368+0000","last_fullsized":"2024-04-26T02:24:35.146368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:16:36.622533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86706520,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":3452,"num_write_kb":123177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.676622+0000","last_change":"2024-04-26T02:21:31.436585+0000","last_active":"2024-04-26T02:24:34.676622+0000","last_peered":"2024-04-26T02:24:34.676622+0000","last_clean":"2024-04-26T02:24:34.676622+0000","last_became_active":"2024-04-26T02:21:31.436226+0000","last_became_peered":"2024-04-26T02:21:31.436226+0000","last_unstale":"2024-04-26T02:24:34.676622+0000","last_undegraded":"2024-04-26T02:24:34.676622+0000","last_fullsized":"2024-04-26T02:24:34.676622+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:21:29.470825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"23'4455","reported_seq":4472,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.147147+0000","last_change":"2024-04-26T02:21:37.494384+0000","last_active":"2024-04-26T02:24:35.147147+0000","last_peered":"2024-04-26T02:24:35.147147+0000","last_clean":"2024-04-26T02:24:35.147147+0000","last_became_active":"2024-04-26T02:21:37.493922+0000","last_became_peered":"2024-04-26T02:21:37.493922+0000","last_unstale":"2024-04-26T02:24:35.147147+0000","last_undegraded":"2024-04-26T02:24:35.147147+0000","last_fullsized":"2024-04-26T02:24:35.147147+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4455,"log_dups_size":0,"ondisk_log_size":4455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:42:04.618288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113312264,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":4436,"num_write_kb":167962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.683586+0000","last_change":"2024-04-26T02:21:31.433313+0000","last_active":"2024-04-26T02:24:34.683586+0000","last_peered":"2024-04-26T02:24:34.683586+0000","last_clean":"2024-04-26T02:24:34.683586+0000","last_became_active":"2024-04-26T02:21:31.432395+0000","last_became_peered":"2024-04-26T02:21:31.432395+0000","last_unstale":"2024-04-26T02:24:34.683586+0000","last_undegraded":"2024-04-26T02:24:34.683586+0000","last_fullsized":"2024-04-26T02:24:34.683586+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:57:44.139176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"23'4930","reported_seq":4959,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:49.161666+0000","last_change":"2024-04-26T02:21:37.485786+0000","last_active":"2024-04-26T02:24:49.161666+0000","last_peered":"2024-04-26T02:24:49.161666+0000","last_clean":"2024-04-26T02:24:49.161666+0000","last_became_active":"2024-04-26T02:21:37.484172+0000","last_became_peered":"2024-04-26T02:21:37.484172+0000","last_unstale":"2024-04-26T02:24:49.161666+0000","last_undegraded":"2024-04-26T02:24:49.161666+0000","last_fullsized":"2024-04-26T02:24:49.161666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:40:57.948908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91030032,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":49,"num_read_kb":40,"num_write":4960,"num_write_kb":192278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.734226+0000","last_change":"2024-04-26T02:21:33.341715+0000","last_active":"2024-04-26T02:24:34.734226+0000","last_peered":"2024-04-26T02:24:34.734226+0000","last_clean":"2024-04-26T02:24:34.734226+0000","last_became_active":"2024-04-26T02:21:33.340937+0000","last_became_peered":"2024-04-26T02:21:33.340937+0000","last_unstale":"2024-04-26T02:24:34.734226+0000","last_undegraded":"2024-04-26T02:24:34.734226+0000","last_fullsized":"2024-04-26T02:24:34.734226+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:03:17.530982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.683452+0000","last_change":"2024-04-26T02:21:29.434185+0000","last_active":"2024-04-26T02:24:34.683452+0000","last_peered":"2024-04-26T02:24:34.683452+0000","last_clean":"2024-04-26T02:24:34.683452+0000","last_became_active":"2024-04-26T02:21:29.432363+0000","last_became_peered":"2024-04-26T02:21:29.432363+0000","last_unstale":"2024-04-26T02:24:34.683452+0000","last_undegraded":"2024-04-26T02:24:34.683452+0000","last_fullsized":"2024-04-26T02:24:34.683452+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:28.375870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:28.375870+0000","last_clean_scrub_stamp":"2024-04-26T02:21:28.375870+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:18:11.058445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"23'4891","reported_seq":4906,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.147573+0000","last_change":"2024-04-26T02:21:37.488171+0000","last_active":"2024-04-26T02:24:35.147573+0000","last_peered":"2024-04-26T02:24:35.147573+0000","last_clean":"2024-04-26T02:24:35.147573+0000","last_became_active":"2024-04-26T02:21:37.486673+0000","last_became_peered":"2024-04-26T02:21:37.486673+0000","last_unstale":"2024-04-26T02:24:35.147573+0000","last_undegraded":"2024-04-26T02:24:35.147573+0000","last_fullsized":"2024-04-26T02:24:35.147573+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4891,"log_dups_size":0,"ondisk_log_size":4891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:29:05.728663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4819,"num_write_kb":226567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"23'9645","reported_seq":9761,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:51.709831+0000","last_change":"2024-04-26T02:21:37.476830+0000","last_active":"2024-04-26T02:24:51.709831+0000","last_peered":"2024-04-26T02:24:51.709831+0000","last_clean":"2024-04-26T02:24:51.709831+0000","last_became_active":"2024-04-26T02:21:37.476072+0000","last_became_peered":"2024-04-26T02:21:37.476072+0000","last_unstale":"2024-04-26T02:24:51.709831+0000","last_undegraded":"2024-04-26T02:24:51.709831+0000","last_fullsized":"2024-04-26T02:24:51.709831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":9645,"log_dups_size":0,"ondisk_log_size":9645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:26:14.037852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138481144,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":348,"num_read_kb":309,"num_write":7857,"num_write_kb":214099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"23'5768","reported_seq":5892,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:53.974613+0000","last_change":"2024-04-26T02:21:37.490171+0000","last_active":"2024-04-26T02:24:53.974613+0000","last_peered":"2024-04-26T02:24:53.974613+0000","last_clean":"2024-04-26T02:24:53.974613+0000","last_became_active":"2024-04-26T02:21:37.488724+0000","last_became_peered":"2024-04-26T02:21:37.488724+0000","last_unstale":"2024-04-26T02:24:53.974613+0000","last_undegraded":"2024-04-26T02:24:53.974613+0000","last_fullsized":"2024-04-26T02:24:53.974613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":5768,"log_dups_size":0,"ondisk_log_size":5768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:17.512625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134480400,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":132,"num_read_kb":124,"num_write":5774,"num_write_kb":201317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"22'3903","reported_seq":3918,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.656121+0000","last_change":"2024-04-26T02:21:37.492466+0000","last_active":"2024-04-26T02:24:34.656121+0000","last_peered":"2024-04-26T02:24:34.656121+0000","last_clean":"2024-04-26T02:24:34.656121+0000","last_became_active":"2024-04-26T02:21:37.491785+0000","last_became_peered":"2024-04-26T02:21:37.491785+0000","last_unstale":"2024-04-26T02:24:34.656121+0000","last_undegraded":"2024-04-26T02:24:34.656121+0000","last_fullsized":"2024-04-26T02:24:34.656121+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3903,"log_dups_size":0,"ondisk_log_size":3903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:44.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88842240,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3901,"num_write_kb":140906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"23'3918","reported_seq":3935,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.141267+0000","last_change":"2024-04-26T02:21:37.493141+0000","last_active":"2024-04-26T02:24:35.141267+0000","last_peered":"2024-04-26T02:24:35.141267+0000","last_clean":"2024-04-26T02:24:35.141267+0000","last_became_active":"2024-04-26T02:21:37.490836+0000","last_became_peered":"2024-04-26T02:21:37.490836+0000","last_unstale":"2024-04-26T02:24:35.141267+0000","last_undegraded":"2024-04-26T02:24:35.141267+0000","last_fullsized":"2024-04-26T02:24:35.141267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:35.359899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100672016,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3880,"num_write_kb":189331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"23'3148","reported_seq":3471,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.011294+0000","last_change":"2024-04-26T02:21:37.501742+0000","last_active":"2024-04-26T02:24:35.011294+0000","last_peered":"2024-04-26T02:24:35.011294+0000","last_clean":"2024-04-26T02:24:35.011294+0000","last_became_active":"2024-04-26T02:21:37.500919+0000","last_became_peered":"2024-04-26T02:21:37.500919+0000","last_unstale":"2024-04-26T02:24:35.011294+0000","last_undegraded":"2024-04-26T02:24:35.011294+0000","last_fullsized":"2024-04-26T02:24:35.011294+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:15:44.491969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84148752,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1028,"num_read_kb":921,"num_write":3342,"num_write_kb":128651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"23'4858","reported_seq":4947,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.232580+0000","last_change":"2024-04-26T02:21:37.497393+0000","last_active":"2024-04-26T02:24:35.232580+0000","last_peered":"2024-04-26T02:24:35.232580+0000","last_clean":"2024-04-26T02:24:35.232580+0000","last_became_active":"2024-04-26T02:21:37.489359+0000","last_became_peered":"2024-04-26T02:21:37.489359+0000","last_unstale":"2024-04-26T02:24:35.232580+0000","last_undegraded":"2024-04-26T02:24:35.232580+0000","last_fullsized":"2024-04-26T02:24:35.232580+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:47.007579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130163216,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":282,"num_read_kb":274,"num_write":5118,"num_write_kb":178670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"23'3713","reported_seq":3728,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.149169+0000","last_change":"2024-04-26T02:21:37.489974+0000","last_active":"2024-04-26T02:24:35.149169+0000","last_peered":"2024-04-26T02:24:35.149169+0000","last_clean":"2024-04-26T02:24:35.149169+0000","last_became_active":"2024-04-26T02:21:37.488144+0000","last_became_peered":"2024-04-26T02:21:37.488144+0000","last_unstale":"2024-04-26T02:24:35.149169+0000","last_undegraded":"2024-04-26T02:24:35.149169+0000","last_fullsized":"2024-04-26T02:24:35.149169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3713,"log_dups_size":0,"ondisk_log_size":3713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:53:06.375314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":168964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"23'3581","reported_seq":3599,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.139594+0000","last_change":"2024-04-26T02:21:37.495244+0000","last_active":"2024-04-26T02:24:35.139594+0000","last_peered":"2024-04-26T02:24:35.139594+0000","last_clean":"2024-04-26T02:24:35.139594+0000","last_became_active":"2024-04-26T02:21:37.494442+0000","last_became_peered":"2024-04-26T02:21:37.494442+0000","last_unstale":"2024-04-26T02:24:35.139594+0000","last_undegraded":"2024-04-26T02:24:35.139594+0000","last_fullsized":"2024-04-26T02:24:35.139594+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3581,"log_dups_size":0,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:36.918805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":3543,"num_write_kb":168836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"22'4310","reported_seq":4325,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.683294+0000","last_change":"2024-04-26T02:21:37.498351+0000","last_active":"2024-04-26T02:24:34.683294+0000","last_peered":"2024-04-26T02:24:34.683294+0000","last_clean":"2024-04-26T02:24:34.683294+0000","last_became_active":"2024-04-26T02:21:37.497599+0000","last_became_peered":"2024-04-26T02:21:37.497599+0000","last_unstale":"2024-04-26T02:24:34.683294+0000","last_undegraded":"2024-04-26T02:24:34.683294+0000","last_fullsized":"2024-04-26T02:24:34.683294+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:05:22.895436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4304,"num_write_kb":209927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"23'4328","reported_seq":4349,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.137635+0000","last_change":"2024-04-26T02:21:37.487437+0000","last_active":"2024-04-26T02:24:35.137635+0000","last_peered":"2024-04-26T02:24:35.137635+0000","last_clean":"2024-04-26T02:24:35.137635+0000","last_became_active":"2024-04-26T02:21:37.486967+0000","last_became_peered":"2024-04-26T02:21:37.486967+0000","last_unstale":"2024-04-26T02:24:35.137635+0000","last_undegraded":"2024-04-26T02:24:35.137635+0000","last_fullsized":"2024-04-26T02:24:35.137635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:50:14.560813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127401984,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23,"num_read_kb":80,"num_write":4288,"num_write_kb":199362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"23'8884","reported_seq":8927,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:47.788241+0000","last_change":"2024-04-26T02:21:37.496905+0000","last_active":"2024-04-26T02:24:47.788241+0000","last_peered":"2024-04-26T02:24:47.788241+0000","last_clean":"2024-04-26T02:24:47.788241+0000","last_became_active":"2024-04-26T02:21:37.488117+0000","last_became_peered":"2024-04-26T02:21:37.488117+0000","last_unstale":"2024-04-26T02:24:47.788241+0000","last_undegraded":"2024-04-26T02:24:47.788241+0000","last_fullsized":"2024-04-26T02:24:47.788241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":8884,"log_dups_size":0,"ondisk_log_size":8884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:55:56.651035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128582144,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":108,"num_read_kb":91,"num_write":8909,"num_write_kb":228167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3568399112,"num_objects":878,"num_object_clones":0,"num_object_copies":1756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":878,"num_whiteouts":0,"num_read":2493,"num_read_kb":2504,"num_write":159711,"num_write_kb":5636573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6797901824,"data_stored":6835447960,"data_compressed":194293810,"data_compressed_allocated":202072064,"data_compressed_original":243892224,"omap_allocated":0,"internal_metadata":0},"log_size":161261,"ondisk_log_size":161261,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11166,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542199,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1412416,"kb_used_data":1038392,"kb_used_omap":1,"kb_used_meta":374014,"kb_avail":92959424,"statfs":{"total":96636764160,"available":95190450176,"internally_reserved":0,"allocated":1063313408,"data_stored":1066332489,"data_compressed":20319008,"data_compressed_allocated":21159936,"data_compressed_original":25022464,"omap_allocated":1590,"internal_metadata":382990794},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542200,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1148328,"kb_used_data":864544,"kb_used_omap":1,"kb_used_meta":283774,"kb_avail":93223512,"statfs":{"total":96636764160,"available":95460876288,"internally_reserved":0,"allocated":885293056,"data_stored":888264065,"data_compressed":21380520,"data_compressed_allocated":22167552,"data_compressed_original":25878528,"omap_allocated":1595,"internal_metadata":290585029},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542199,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1944992,"kb_used_data":1386328,"kb_used_omap":1,"kb_used_meta":558654,"kb_avail":92426848,"statfs":{"total":96636764160,"available":94645092352,"internally_reserved":0,"allocated":1419599872,"data_stored":1427702961,"data_compressed":38724772,"data_compressed_allocated":40361984,"data_compressed_original":49508352,"omap_allocated":1595,"internal_metadata":572062149},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542201,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1446344,"kb_used_data":1076480,"kb_used_omap":1,"kb_used_meta":369854,"kb_avail":92925496,"statfs":{"total":96636764160,"available":95155707904,"internally_reserved":0,"allocated":1102315520,"data_stored":1108245281,"data_compressed":31089397,"data_compressed_allocated":32296960,"data_compressed_original":39075840,"omap_allocated":1592,"internal_metadata":378730952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542199,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":885764,"kb_used_data":615868,"kb_used_omap":1,"kb_used_meta":269886,"kb_avail":93486076,"statfs":{"total":96636764160,"available":95729741824,"internally_reserved":0,"allocated":630648832,"data_stored":637297197,"data_compressed":27771469,"data_compressed_allocated":28930048,"data_compressed_original":36147200,"omap_allocated":1595,"internal_metadata":276363717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542199,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1387388,"kb_used_data":1018932,"kb_used_omap":1,"kb_used_meta":368446,"kb_avail":92984452,"statfs":{"total":96636764160,"available":95216078848,"internally_reserved":0,"allocated":1043386368,"data_stored":1049606709,"data_compressed":28674963,"data_compressed_allocated":29855744,"data_compressed_original":36777984,"omap_allocated":1595,"internal_metadata":377289157},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464488,"kb_used_data":343072,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93907352,"statfs":{"total":96636764160,"available":96161128448,"internally_reserved":0,"allocated":351305728,"data_stored":352436329,"data_compressed":9222947,"data_compressed_allocated":9637888,"data_compressed_original":11067392,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466656,"kb_used_data":296728,"kb_used_omap":1,"kb_used_meta":169918,"kb_avail":93905184,"statfs":{"total":96636764160,"available":96158908416,"internally_reserved":0,"allocated":303849472,"data_stored":307208889,"data_compressed":17121900,"data_compressed_allocated":17727488,"data_compressed_original":21438464,"omap_allocated":1595,"internal_metadata":173996485},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1043120128,"data_stored":1049004268,"data_compressed":28669380,"data_compressed_allocated":29822976,"data_compressed_original":36265984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":630435840,"data_stored":637223684,"data_compressed":27771469,"data_compressed_allocated":28930048,"data_compressed_original":36147200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303636480,"data_stored":307135376,"data_compressed":17121900,"data_compressed_allocated":17727488,"data_compressed_original":21438464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1102102528,"data_stored":1108171768,"data_compressed":31089397,"data_compressed_allocated":32296960,"data_compressed_original":39075840,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1419386880,"data_stored":1427629448,"data_compressed":38724772,"data_compressed_allocated":40361984,"data_compressed_original":49508352,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":885080064,"data_stored":888190552,"data_compressed":21380520,"data_compressed_allocated":22167552,"data_compressed_original":25878528,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":351092736,"data_stored":352362816,"data_compressed":9222947,"data_compressed_allocated":9637888,"data_compressed_original":11067392,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1063047168,"data_stored":1065730048,"data_compressed":20313425,"data_compressed_allocated":21127168,"data_compressed_original":24510464,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:51.578 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:51.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4ef7012640 1 Processor -- start 2024-04-26T02:25:51.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4ef7012640 1 -- start start 2024-04-26T02:25:51.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4ef7012640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ef00ab2e0 con 0x7f4ef01447d0 2024-04-26T02:25:51.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4ef7012640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ef00a9c90 con 0x7f4ef0143cf0 2024-04-26T02:25:51.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4ef7012640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ef01440d0 con 0x7f4ef013dbe0 2024-04-26T02:25:51.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4eeffff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f4ef01447d0 0x7f4ef006f0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51738/0 (socket says 172.21.15.87:51738) 2024-04-26T02:25:51.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4eeffff640 1 -- 172.21.15.87:0/141653412 learned_addr learned my addr 172.21.15.87:0/141653412 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:51.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4eef7fe640 1 -- 172.21.15.87:0/141653412 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 367076227 0 0) 0x7f4ef00ab2e0 con 0x7f4ef01447d0 2024-04-26T02:25:51.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4eef7fe640 1 -- 172.21.15.87:0/141653412 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ee0003530 con 0x7f4ef01447d0 2024-04-26T02:25:51.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4eef7fe640 1 -- 172.21.15.87:0/141653412 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1173690046 0 0) 0x7f4ef01440d0 con 0x7f4ef013dbe0 2024-04-26T02:25:51.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4eef7fe640 1 -- 172.21.15.87:0/141653412 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ef00ab2e0 con 0x7f4ef013dbe0 2024-04-26T02:25:51.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4eef7fe640 1 -- 172.21.15.87:0/141653412 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3059782489 0 0) 0x7f4ee0003530 con 0x7f4ef01447d0 2024-04-26T02:25:51.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4eef7fe640 1 -- 172.21.15.87:0/141653412 >> v1:172.21.15.87:6790/0 conn(0x7f4ef013dbe0 legacy=0x7f4ef014be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:51.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4eef7fe640 1 -- 172.21.15.87:0/141653412 >> v1:172.21.15.122:6789/0 conn(0x7f4ef0143cf0 legacy=0x7f4ef013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:51.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4eef7fe640 1 -- 172.21.15.87:0/141653412 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef00a9340 con 0x7f4ef01447d0 2024-04-26T02:25:51.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4eef7fe640 1 -- 172.21.15.87:0/141653412 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f4edc001620 con 0x7f4ef01447d0 2024-04-26T02:25:51.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4eef7fe640 1 -- 172.21.15.87:0/141653412 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4edc001890 con 0x7f4ef01447d0 2024-04-26T02:25:51.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.696+0000 7f4eef7fe640 1 -- 172.21.15.87:0/141653412 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f4edc001c60 con 0x7f4ef01447d0 2024-04-26T02:25:51.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4ef7012640 1 -- 172.21.15.87:0/141653412 >> v1:172.21.15.87:6789/0 conn(0x7f4ef01447d0 legacy=0x7f4ef006f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:51.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4ef7012640 1 -- 172.21.15.87:0/141653412 shutdown_connections 2024-04-26T02:25:51.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4ef7012640 1 -- 172.21.15.87:0/141653412 >> 172.21.15.87:0/141653412 conn(0x7f4ef00b40d0 msgr2=0x7f4ef00b44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:51.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4ef7012640 1 -- 172.21.15.87:0/141653412 shutdown_connections 2024-04-26T02:25:51.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4ef7012640 1 -- 172.21.15.87:0/141653412 wait complete. 2024-04-26T02:25:51.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4ef7012640 1 Processor -- start 2024-04-26T02:25:51.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4ef7012640 1 -- start start 2024-04-26T02:25:51.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4ef7012640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ef0132b30 con 0x7f4ef013dbe0 2024-04-26T02:25:51.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4ef7012640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ef01403d0 con 0x7f4ef0135720 2024-04-26T02:25:51.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4ef7012640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ef01406b0 con 0x7f4ef0143cf0 2024-04-26T02:25:51.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4ef4dae640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f4ef013dbe0 0x7f4ef01db130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51746/0 (socket says 172.21.15.87:51746) 2024-04-26T02:25:51.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4ef4dae640 1 -- 172.21.15.87:0/3131530588 learned_addr learned my addr 172.21.15.87:0/3131530588 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:51.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2449464837 0 0) 0x7f4ef01406b0 con 0x7f4ef0143cf0 2024-04-26T02:25:51.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ed4003530 con 0x7f4ef0143cf0 2024-04-26T02:25:51.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2496222872 0 0) 0x7f4ef0132b30 con 0x7f4ef013dbe0 2024-04-26T02:25:51.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ef01406b0 con 0x7f4ef013dbe0 2024-04-26T02:25:51.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3912629426 0 0) 0x7f4ed4003530 con 0x7f4ef0143cf0 2024-04-26T02:25:51.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ef0132b30 con 0x7f4ef0143cf0 2024-04-26T02:25:51.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f4edc001be0 con 0x7f4ef0143cf0 2024-04-26T02:25:51.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 922604552 0 0) 0x7f4ef01406b0 con 0x7f4ef013dbe0 2024-04-26T02:25:51.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ed4003530 con 0x7f4ef013dbe0 2024-04-26T02:25:51.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f4ee4001900 con 0x7f4ef013dbe0 2024-04-26T02:25:51.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1066778145 0 0) 0x7f4ef01403d0 con 0x7f4ef0135720 2024-04-26T02:25:51.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ef01406b0 con 0x7f4ef0135720 2024-04-26T02:25:51.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4279776630 0 0) 0x7f4ef0132b30 con 0x7f4ef0143cf0 2024-04-26T02:25:51.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 >> v1:172.21.15.122:6789/0 conn(0x7f4ef0135720 legacy=0x7f4ef0132420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:51.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 >> v1:172.21.15.87:6789/0 conn(0x7f4ef013dbe0 legacy=0x7f4ef01db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:51.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef0141290 con 0x7f4ef0143cf0 2024-04-26T02:25:51.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4ef7012640 1 -- 172.21.15.87:0/3131530588 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef00afd70 con 0x7f4ef0143cf0 2024-04-26T02:25:51.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4ef7012640 1 -- 172.21.15.87:0/3131530588 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4ef01db840 con 0x7f4ef0143cf0 2024-04-26T02:25:51.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4edc001890 con 0x7f4ef0143cf0 2024-04-26T02:25:51.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f4edc004bb0 con 0x7f4ef0143cf0 2024-04-26T02:25:51.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.700+0000 7f4ef7012640 1 -- 172.21.15.87:0/3131530588 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ed0005090 con 0x7f4ef0143cf0 2024-04-26T02:25:51.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.704+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f4edc01a2d0 con 0x7f4ef0143cf0 2024-04-26T02:25:51.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.704+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f4edc081280 con 0x7f4ef0143cf0 2024-04-26T02:25:51.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.708+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f4edc052130 con 0x7f4ef0143cf0 2024-04-26T02:25:51.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.864+0000 7f4ef7012640 1 -- 172.21.15.87:0/3131530588 --> v1:172.21.15.122:6816/416193625 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4ed0000ba0 con 0x7f4ed4068470 2024-04-26T02:25:51.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.872+0000 7f4eed7fa640 1 -- 172.21.15.87:0/3131530588 <== mgr.4108 v1:172.21.15.122:6816/416193625 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+117694 (unknown 2965378022 0 2277588350) 0x7f4ed0000ba0 con 0x7f4ed4068470 2024-04-26T02:25:51.876 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:25:51.877 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-26T02:25:51.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.876+0000 7f4ef7012640 1 -- 172.21.15.87:0/3131530588 >> v1:172.21.15.122:6816/416193625 conn(0x7f4ed4068470 legacy=0x7f4ed406a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:51.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.876+0000 7f4ef7012640 1 -- 172.21.15.87:0/3131530588 >> v1:172.21.15.87:6790/0 conn(0x7f4ef0143cf0 legacy=0x7f4ef0131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:51.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.876+0000 7f4ef7012640 1 -- 172.21.15.87:0/3131530588 shutdown_connections 2024-04-26T02:25:51.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.876+0000 7f4ef7012640 1 -- 172.21.15.87:0/3131530588 >> 172.21.15.87:0/3131530588 conn(0x7f4ef00b40d0 msgr2=0x7f4ef0074c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:51.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.876+0000 7f4ef7012640 1 -- 172.21.15.87:0/3131530588 shutdown_connections 2024-04-26T02:25:51.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:51.876+0000 7f4ef7012640 1 -- 172.21.15.87:0/3131530588 wait complete. 2024-04-26T02:25:51.894 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-04-26T02:25:51.415976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3568989480,"num_objects":880,"num_object_clones":0,"num_object_copies":1760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":2609,"num_read_kb":2604,"num_write":159870,"num_write_kb":5638989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161378,"ondisk_log_size":161378,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":9156376,"kb_used_data":6640344,"kb_used_omap":12,"kb_used_meta":2515955,"kb_avail":745818344,"statfs":{"total":773094113280,"available":763717984256,"internally_reserved":0,"allocated":6799712256,"data_stored":6837093920,"data_compressed":194304976,"data_compressed_allocated":202137600,"data_compressed_original":244916224,"omap_allocated":12751,"internal_metadata":2576338481},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36864,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003159"},"pg_stats":[{"pgid":"3.1f","version":"22'2749","reported_seq":2768,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.658393+0000","last_change":"2024-04-26T02:21:37.492661+0000","last_active":"2024-04-26T02:24:34.658393+0000","last_peered":"2024-04-26T02:24:34.658393+0000","last_clean":"2024-04-26T02:24:34.658393+0000","last_became_active":"2024-04-26T02:21:37.491078+0000","last_became_peered":"2024-04-26T02:21:37.491078+0000","last_unstale":"2024-04-26T02:24:34.658393+0000","last_undegraded":"2024-04-26T02:24:34.658393+0000","last_fullsized":"2024-04-26T02:24:34.658393+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":2749,"log_dups_size":0,"ondisk_log_size":2749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:56:51.567611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":71,"num_write":2855,"num_write_kb":123586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"23'6576","reported_seq":6592,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:42.266530+0000","last_change":"2024-04-26T02:21:37.485093+0000","last_active":"2024-04-26T02:24:42.266530+0000","last_peered":"2024-04-26T02:24:42.266530+0000","last_clean":"2024-04-26T02:24:42.266530+0000","last_became_active":"2024-04-26T02:21:37.484287+0000","last_became_peered":"2024-04-26T02:21:37.484287+0000","last_unstale":"2024-04-26T02:24:42.266530+0000","last_undegraded":"2024-04-26T02:24:42.266530+0000","last_fullsized":"2024-04-26T02:24:42.266530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":6576,"log_dups_size":0,"ondisk_log_size":6576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:56:08.755136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6549,"num_write_kb":206212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"23'5724","reported_seq":5759,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:47.550994+0000","last_change":"2024-04-26T02:21:37.498519+0000","last_active":"2024-04-26T02:24:47.550994+0000","last_peered":"2024-04-26T02:24:47.550994+0000","last_clean":"2024-04-26T02:24:47.550994+0000","last_became_active":"2024-04-26T02:21:37.497700+0000","last_became_peered":"2024-04-26T02:21:37.497700+0000","last_unstale":"2024-04-26T02:24:47.550994+0000","last_undegraded":"2024-04-26T02:24:47.550994+0000","last_fullsized":"2024-04-26T02:24:47.550994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":5724,"log_dups_size":0,"ondisk_log_size":5724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:02:58.912915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":76,"num_read_kb":124,"num_write":5736,"num_write_kb":155565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"22'4059","reported_seq":4077,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.734233+0000","last_change":"2024-04-26T02:21:37.497650+0000","last_active":"2024-04-26T02:24:34.734233+0000","last_peered":"2024-04-26T02:24:34.734233+0000","last_clean":"2024-04-26T02:24:34.734233+0000","last_became_active":"2024-04-26T02:21:37.496863+0000","last_became_peered":"2024-04-26T02:21:37.496863+0000","last_unstale":"2024-04-26T02:24:34.734233+0000","last_undegraded":"2024-04-26T02:24:34.734233+0000","last_fullsized":"2024-04-26T02:24:34.734233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4059,"log_dups_size":0,"ondisk_log_size":4059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:57:07.038484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":4059,"num_write_kb":185415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"23'3948","reported_seq":3972,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.141201+0000","last_change":"2024-04-26T02:21:37.491890+0000","last_active":"2024-04-26T02:24:35.141201+0000","last_peered":"2024-04-26T02:24:35.141201+0000","last_clean":"2024-04-26T02:24:35.141201+0000","last_became_active":"2024-04-26T02:21:37.490474+0000","last_became_peered":"2024-04-26T02:21:37.490474+0000","last_unstale":"2024-04-26T02:24:35.141201+0000","last_undegraded":"2024-04-26T02:24:35.141201+0000","last_fullsized":"2024-04-26T02:24:35.141201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3948,"log_dups_size":0,"ondisk_log_size":3948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:56.993069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120324096,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":34,"num_read_kb":151,"num_write":3926,"num_write_kb":190983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"23'6081","reported_seq":6102,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:41.034089+0000","last_change":"2024-04-26T02:22:35.114133+0000","last_active":"2024-04-26T02:24:41.034089+0000","last_peered":"2024-04-26T02:24:41.034089+0000","last_clean":"2024-04-26T02:24:41.034089+0000","last_became_active":"2024-04-26T02:21:37.485110+0000","last_became_peered":"2024-04-26T02:21:37.485110+0000","last_unstale":"2024-04-26T02:24:41.034089+0000","last_undegraded":"2024-04-26T02:24:41.034089+0000","last_fullsized":"2024-04-26T02:24:41.034089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":6081,"log_dups_size":0,"ondisk_log_size":6081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:39:18.389527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":6064,"num_write_kb":187005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"23'5039","reported_seq":5073,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:52.103217+0000","last_change":"2024-04-26T02:21:37.489148+0000","last_active":"2024-04-26T02:24:52.103217+0000","last_peered":"2024-04-26T02:24:52.103217+0000","last_clean":"2024-04-26T02:24:52.103217+0000","last_became_active":"2024-04-26T02:21:37.488519+0000","last_became_peered":"2024-04-26T02:21:37.488519+0000","last_unstale":"2024-04-26T02:24:52.103217+0000","last_undegraded":"2024-04-26T02:24:52.103217+0000","last_fullsized":"2024-04-26T02:24:52.103217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:41.178413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117441024,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":5034,"num_write_kb":181716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"23'7153","reported_seq":7190,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:52.126229+0000","last_change":"2024-04-26T02:21:37.496383+0000","last_active":"2024-04-26T02:24:52.126229+0000","last_peered":"2024-04-26T02:24:52.126229+0000","last_clean":"2024-04-26T02:24:52.126229+0000","last_became_active":"2024-04-26T02:21:37.495694+0000","last_became_peered":"2024-04-26T02:21:37.495694+0000","last_unstale":"2024-04-26T02:24:52.126229+0000","last_undegraded":"2024-04-26T02:24:52.126229+0000","last_fullsized":"2024-04-26T02:24:52.126229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":7153,"log_dups_size":0,"ondisk_log_size":7153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:27:30.478083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023936,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":80,"num_read_kb":65,"num_write":7184,"num_write_kb":197893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"23'8281","reported_seq":8304,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:25:39.938993+0000","last_change":"2024-04-26T02:21:37.490841+0000","last_active":"2024-04-26T02:25:39.938993+0000","last_peered":"2024-04-26T02:25:39.938993+0000","last_clean":"2024-04-26T02:25:39.938993+0000","last_became_active":"2024-04-26T02:21:37.490119+0000","last_became_peered":"2024-04-26T02:21:37.490119+0000","last_unstale":"2024-04-26T02:25:39.938993+0000","last_undegraded":"2024-04-26T02:25:39.938993+0000","last_fullsized":"2024-04-26T02:25:39.938993+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":8281,"log_dups_size":0,"ondisk_log_size":8281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:36:46.649805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137060352,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":8291,"num_write_kb":229215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"23'3756","reported_seq":3771,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.020013+0000","last_change":"2024-04-26T02:21:37.507074+0000","last_active":"2024-04-26T02:24:35.020013+0000","last_peered":"2024-04-26T02:24:35.020013+0000","last_clean":"2024-04-26T02:24:35.020013+0000","last_became_active":"2024-04-26T02:21:37.506309+0000","last_became_peered":"2024-04-26T02:21:37.506309+0000","last_unstale":"2024-04-26T02:24:35.020013+0000","last_undegraded":"2024-04-26T02:24:35.020013+0000","last_fullsized":"2024-04-26T02:24:35.020013+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3756,"log_dups_size":0,"ondisk_log_size":3756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:49:23.247821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3724,"num_write_kb":119684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"23'7177","reported_seq":7192,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:50.387812+0000","last_change":"2024-04-26T02:21:37.487697+0000","last_active":"2024-04-26T02:24:50.387812+0000","last_peered":"2024-04-26T02:24:50.387812+0000","last_clean":"2024-04-26T02:24:50.387812+0000","last_became_active":"2024-04-26T02:21:37.486868+0000","last_became_peered":"2024-04-26T02:21:37.486868+0000","last_unstale":"2024-04-26T02:24:50.387812+0000","last_undegraded":"2024-04-26T02:24:50.387812+0000","last_fullsized":"2024-04-26T02:24:50.387812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":7177,"log_dups_size":0,"ondisk_log_size":7177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:26:13.460721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7176,"num_write_kb":198373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"23'4534","reported_seq":4549,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:44.307401+0000","last_change":"2024-04-26T02:21:37.488921+0000","last_active":"2024-04-26T02:24:44.307401+0000","last_peered":"2024-04-26T02:24:44.307401+0000","last_clean":"2024-04-26T02:24:44.307401+0000","last_became_active":"2024-04-26T02:21:37.487980+0000","last_became_peered":"2024-04-26T02:21:37.487980+0000","last_unstale":"2024-04-26T02:24:44.307401+0000","last_undegraded":"2024-04-26T02:24:44.307401+0000","last_fullsized":"2024-04-26T02:24:44.307401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4534,"log_dups_size":0,"ondisk_log_size":4534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:10.065671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4488,"num_write_kb":107395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.730342+0000","last_change":"2024-04-26T02:21:31.429340+0000","last_active":"2024-04-26T02:24:34.730342+0000","last_peered":"2024-04-26T02:24:34.730342+0000","last_clean":"2024-04-26T02:24:34.730342+0000","last_became_active":"2024-04-26T02:21:31.428657+0000","last_became_peered":"2024-04-26T02:21:31.428657+0000","last_unstale":"2024-04-26T02:24:34.730342+0000","last_undegraded":"2024-04-26T02:24:34.730342+0000","last_fullsized":"2024-04-26T02:24:34.730342+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:07:12.674824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'4886","reported_seq":4909,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:48.178325+0000","last_change":"2024-04-26T02:21:37.482817+0000","last_active":"2024-04-26T02:24:48.178325+0000","last_peered":"2024-04-26T02:24:48.178325+0000","last_clean":"2024-04-26T02:24:48.178325+0000","last_became_active":"2024-04-26T02:21:37.481620+0000","last_became_peered":"2024-04-26T02:21:37.481620+0000","last_unstale":"2024-04-26T02:24:48.178325+0000","last_undegraded":"2024-04-26T02:24:48.178325+0000","last_fullsized":"2024-04-26T02:24:48.178325+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4886,"log_dups_size":0,"ondisk_log_size":4886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:26.760929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96600064,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":4875,"num_write_kb":131621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.658277+0000","last_change":"2024-04-26T02:21:31.432330+0000","last_active":"2024-04-26T02:24:34.658277+0000","last_peered":"2024-04-26T02:24:34.658277+0000","last_clean":"2024-04-26T02:24:34.658277+0000","last_became_active":"2024-04-26T02:21:31.431958+0000","last_became_peered":"2024-04-26T02:21:31.431958+0000","last_unstale":"2024-04-26T02:24:34.658277+0000","last_undegraded":"2024-04-26T02:24:34.658277+0000","last_fullsized":"2024-04-26T02:24:34.658277+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:59:02.906743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'4222","reported_seq":4237,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.143067+0000","last_change":"2024-04-26T02:21:37.488836+0000","last_active":"2024-04-26T02:24:35.143067+0000","last_peered":"2024-04-26T02:24:35.143067+0000","last_clean":"2024-04-26T02:24:35.143067+0000","last_became_active":"2024-04-26T02:21:37.487980+0000","last_became_peered":"2024-04-26T02:21:37.487980+0000","last_unstale":"2024-04-26T02:24:35.143067+0000","last_undegraded":"2024-04-26T02:24:35.143067+0000","last_fullsized":"2024-04-26T02:24:35.143067+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:07:49.262710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4175,"num_write_kb":201729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.684259+0000","last_change":"2024-04-26T02:21:31.430988+0000","last_active":"2024-04-26T02:24:34.684259+0000","last_peered":"2024-04-26T02:24:34.684259+0000","last_clean":"2024-04-26T02:24:34.684259+0000","last_became_active":"2024-04-26T02:21:31.430227+0000","last_became_peered":"2024-04-26T02:21:31.430227+0000","last_unstale":"2024-04-26T02:24:34.684259+0000","last_undegraded":"2024-04-26T02:24:34.684259+0000","last_fullsized":"2024-04-26T02:24:34.684259+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:32:21.265195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'6121","reported_seq":6142,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:54.144536+0000","last_change":"2024-04-26T02:21:37.483474+0000","last_active":"2024-04-26T02:24:54.144536+0000","last_peered":"2024-04-26T02:24:54.144536+0000","last_clean":"2024-04-26T02:24:54.144536+0000","last_became_active":"2024-04-26T02:21:37.482894+0000","last_became_peered":"2024-04-26T02:21:37.482894+0000","last_unstale":"2024-04-26T02:24:54.144536+0000","last_undegraded":"2024-04-26T02:24:54.144536+0000","last_fullsized":"2024-04-26T02:24:54.144536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":6121,"log_dups_size":0,"ondisk_log_size":6121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:14.817911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99354976,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":6134,"num_write_kb":151141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.659060+0000","last_change":"2024-04-26T02:21:31.431358+0000","last_active":"2024-04-26T02:24:34.659060+0000","last_peered":"2024-04-26T02:24:34.659060+0000","last_clean":"2024-04-26T02:24:34.659060+0000","last_became_active":"2024-04-26T02:21:31.431071+0000","last_became_peered":"2024-04-26T02:21:31.431071+0000","last_unstale":"2024-04-26T02:24:34.659060+0000","last_undegraded":"2024-04-26T02:24:34.659060+0000","last_fullsized":"2024-04-26T02:24:34.659060+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:41:07.918405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"22'4180","reported_seq":4225,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.735876+0000","last_change":"2024-04-26T02:21:37.490970+0000","last_active":"2024-04-26T02:24:34.735876+0000","last_peered":"2024-04-26T02:24:34.735876+0000","last_clean":"2024-04-26T02:24:34.735876+0000","last_became_active":"2024-04-26T02:21:37.490105+0000","last_became_peered":"2024-04-26T02:21:37.490105+0000","last_unstale":"2024-04-26T02:24:34.735876+0000","last_undegraded":"2024-04-26T02:24:34.735876+0000","last_fullsized":"2024-04-26T02:24:34.735876+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4180,"log_dups_size":0,"ondisk_log_size":4180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:17.335251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121896960,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":122,"num_read_kb":98,"num_write":4227,"num_write_kb":167240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"22'2962","reported_seq":2980,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.670117+0000","last_change":"2024-04-26T02:21:37.494426+0000","last_active":"2024-04-26T02:24:34.670117+0000","last_peered":"2024-04-26T02:24:34.670117+0000","last_clean":"2024-04-26T02:24:34.670117+0000","last_became_active":"2024-04-26T02:21:37.491249+0000","last_became_peered":"2024-04-26T02:21:37.491249+0000","last_unstale":"2024-04-26T02:24:34.670117+0000","last_undegraded":"2024-04-26T02:24:34.670117+0000","last_fullsized":"2024-04-26T02:24:34.670117+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:22:39.029300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74842112,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":7,"num_write":2966,"num_write_kb":163586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.734132+0000","last_change":"2024-04-26T02:21:32.425721+0000","last_active":"2024-04-26T02:24:34.734132+0000","last_peered":"2024-04-26T02:24:34.734132+0000","last_clean":"2024-04-26T02:24:34.734132+0000","last_became_active":"2024-04-26T02:21:32.425189+0000","last_became_peered":"2024-04-26T02:21:32.425189+0000","last_unstale":"2024-04-26T02:24:34.734132+0000","last_undegraded":"2024-04-26T02:24:34.734132+0000","last_fullsized":"2024-04-26T02:24:34.734132+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:56:47.156837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"23'3481","reported_seq":3503,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.146368+0000","last_change":"2024-04-26T02:21:37.486201+0000","last_active":"2024-04-26T02:24:35.146368+0000","last_peered":"2024-04-26T02:24:35.146368+0000","last_clean":"2024-04-26T02:24:35.146368+0000","last_became_active":"2024-04-26T02:21:37.485432+0000","last_became_peered":"2024-04-26T02:21:37.485432+0000","last_unstale":"2024-04-26T02:24:35.146368+0000","last_undegraded":"2024-04-26T02:24:35.146368+0000","last_fullsized":"2024-04-26T02:24:35.146368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:16:36.622533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86706520,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":3452,"num_write_kb":123177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.676622+0000","last_change":"2024-04-26T02:21:31.436585+0000","last_active":"2024-04-26T02:24:34.676622+0000","last_peered":"2024-04-26T02:24:34.676622+0000","last_clean":"2024-04-26T02:24:34.676622+0000","last_became_active":"2024-04-26T02:21:31.436226+0000","last_became_peered":"2024-04-26T02:21:31.436226+0000","last_unstale":"2024-04-26T02:24:34.676622+0000","last_undegraded":"2024-04-26T02:24:34.676622+0000","last_fullsized":"2024-04-26T02:24:34.676622+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:21:29.470825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"23'4455","reported_seq":4472,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.147147+0000","last_change":"2024-04-26T02:21:37.494384+0000","last_active":"2024-04-26T02:24:35.147147+0000","last_peered":"2024-04-26T02:24:35.147147+0000","last_clean":"2024-04-26T02:24:35.147147+0000","last_became_active":"2024-04-26T02:21:37.493922+0000","last_became_peered":"2024-04-26T02:21:37.493922+0000","last_unstale":"2024-04-26T02:24:35.147147+0000","last_undegraded":"2024-04-26T02:24:35.147147+0000","last_fullsized":"2024-04-26T02:24:35.147147+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4455,"log_dups_size":0,"ondisk_log_size":4455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:42:04.618288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113312264,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":4436,"num_write_kb":167962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.683586+0000","last_change":"2024-04-26T02:21:31.433313+0000","last_active":"2024-04-26T02:24:34.683586+0000","last_peered":"2024-04-26T02:24:34.683586+0000","last_clean":"2024-04-26T02:24:34.683586+0000","last_became_active":"2024-04-26T02:21:31.432395+0000","last_became_peered":"2024-04-26T02:21:31.432395+0000","last_unstale":"2024-04-26T02:24:34.683586+0000","last_undegraded":"2024-04-26T02:24:34.683586+0000","last_fullsized":"2024-04-26T02:24:34.683586+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:57:44.139176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"23'4930","reported_seq":4959,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:49.161666+0000","last_change":"2024-04-26T02:21:37.485786+0000","last_active":"2024-04-26T02:24:49.161666+0000","last_peered":"2024-04-26T02:24:49.161666+0000","last_clean":"2024-04-26T02:24:49.161666+0000","last_became_active":"2024-04-26T02:21:37.484172+0000","last_became_peered":"2024-04-26T02:21:37.484172+0000","last_unstale":"2024-04-26T02:24:49.161666+0000","last_undegraded":"2024-04-26T02:24:49.161666+0000","last_fullsized":"2024-04-26T02:24:49.161666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:40:57.948908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91030032,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":49,"num_read_kb":40,"num_write":4960,"num_write_kb":192278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.734226+0000","last_change":"2024-04-26T02:21:33.341715+0000","last_active":"2024-04-26T02:24:34.734226+0000","last_peered":"2024-04-26T02:24:34.734226+0000","last_clean":"2024-04-26T02:24:34.734226+0000","last_became_active":"2024-04-26T02:21:33.340937+0000","last_became_peered":"2024-04-26T02:21:33.340937+0000","last_unstale":"2024-04-26T02:24:34.734226+0000","last_undegraded":"2024-04-26T02:24:34.734226+0000","last_fullsized":"2024-04-26T02:24:34.734226+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:03:17.530982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.683452+0000","last_change":"2024-04-26T02:21:29.434185+0000","last_active":"2024-04-26T02:24:34.683452+0000","last_peered":"2024-04-26T02:24:34.683452+0000","last_clean":"2024-04-26T02:24:34.683452+0000","last_became_active":"2024-04-26T02:21:29.432363+0000","last_became_peered":"2024-04-26T02:21:29.432363+0000","last_unstale":"2024-04-26T02:24:34.683452+0000","last_undegraded":"2024-04-26T02:24:34.683452+0000","last_fullsized":"2024-04-26T02:24:34.683452+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:28.375870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:28.375870+0000","last_clean_scrub_stamp":"2024-04-26T02:21:28.375870+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:18:11.058445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"23'4891","reported_seq":4906,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.147573+0000","last_change":"2024-04-26T02:21:37.488171+0000","last_active":"2024-04-26T02:24:35.147573+0000","last_peered":"2024-04-26T02:24:35.147573+0000","last_clean":"2024-04-26T02:24:35.147573+0000","last_became_active":"2024-04-26T02:21:37.486673+0000","last_became_peered":"2024-04-26T02:21:37.486673+0000","last_unstale":"2024-04-26T02:24:35.147573+0000","last_undegraded":"2024-04-26T02:24:35.147573+0000","last_fullsized":"2024-04-26T02:24:35.147573+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4891,"log_dups_size":0,"ondisk_log_size":4891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:29:05.728663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4819,"num_write_kb":226567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"23'9645","reported_seq":9761,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:51.709831+0000","last_change":"2024-04-26T02:21:37.476830+0000","last_active":"2024-04-26T02:24:51.709831+0000","last_peered":"2024-04-26T02:24:51.709831+0000","last_clean":"2024-04-26T02:24:51.709831+0000","last_became_active":"2024-04-26T02:21:37.476072+0000","last_became_peered":"2024-04-26T02:21:37.476072+0000","last_unstale":"2024-04-26T02:24:51.709831+0000","last_undegraded":"2024-04-26T02:24:51.709831+0000","last_fullsized":"2024-04-26T02:24:51.709831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":9645,"log_dups_size":0,"ondisk_log_size":9645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:26:14.037852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138481144,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":348,"num_read_kb":309,"num_write":7857,"num_write_kb":214099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"23'5768","reported_seq":5892,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:53.974613+0000","last_change":"2024-04-26T02:21:37.490171+0000","last_active":"2024-04-26T02:24:53.974613+0000","last_peered":"2024-04-26T02:24:53.974613+0000","last_clean":"2024-04-26T02:24:53.974613+0000","last_became_active":"2024-04-26T02:21:37.488724+0000","last_became_peered":"2024-04-26T02:21:37.488724+0000","last_unstale":"2024-04-26T02:24:53.974613+0000","last_undegraded":"2024-04-26T02:24:53.974613+0000","last_fullsized":"2024-04-26T02:24:53.974613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":5768,"log_dups_size":0,"ondisk_log_size":5768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:17.512625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134480400,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":132,"num_read_kb":124,"num_write":5774,"num_write_kb":201317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"22'3903","reported_seq":3918,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.656121+0000","last_change":"2024-04-26T02:21:37.492466+0000","last_active":"2024-04-26T02:24:34.656121+0000","last_peered":"2024-04-26T02:24:34.656121+0000","last_clean":"2024-04-26T02:24:34.656121+0000","last_became_active":"2024-04-26T02:21:37.491785+0000","last_became_peered":"2024-04-26T02:21:37.491785+0000","last_unstale":"2024-04-26T02:24:34.656121+0000","last_undegraded":"2024-04-26T02:24:34.656121+0000","last_fullsized":"2024-04-26T02:24:34.656121+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3903,"log_dups_size":0,"ondisk_log_size":3903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:44.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88842240,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3901,"num_write_kb":140906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"23'3918","reported_seq":3935,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.141267+0000","last_change":"2024-04-26T02:21:37.493141+0000","last_active":"2024-04-26T02:24:35.141267+0000","last_peered":"2024-04-26T02:24:35.141267+0000","last_clean":"2024-04-26T02:24:35.141267+0000","last_became_active":"2024-04-26T02:21:37.490836+0000","last_became_peered":"2024-04-26T02:21:37.490836+0000","last_unstale":"2024-04-26T02:24:35.141267+0000","last_undegraded":"2024-04-26T02:24:35.141267+0000","last_fullsized":"2024-04-26T02:24:35.141267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:35.359899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100672016,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3880,"num_write_kb":189331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"23'3148","reported_seq":3471,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.011294+0000","last_change":"2024-04-26T02:21:37.501742+0000","last_active":"2024-04-26T02:24:35.011294+0000","last_peered":"2024-04-26T02:24:35.011294+0000","last_clean":"2024-04-26T02:24:35.011294+0000","last_became_active":"2024-04-26T02:21:37.500919+0000","last_became_peered":"2024-04-26T02:21:37.500919+0000","last_unstale":"2024-04-26T02:24:35.011294+0000","last_undegraded":"2024-04-26T02:24:35.011294+0000","last_fullsized":"2024-04-26T02:24:35.011294+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:15:44.491969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84148752,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1028,"num_read_kb":921,"num_write":3342,"num_write_kb":128651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"23'4858","reported_seq":4947,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.232580+0000","last_change":"2024-04-26T02:21:37.497393+0000","last_active":"2024-04-26T02:24:35.232580+0000","last_peered":"2024-04-26T02:24:35.232580+0000","last_clean":"2024-04-26T02:24:35.232580+0000","last_became_active":"2024-04-26T02:21:37.489359+0000","last_became_peered":"2024-04-26T02:21:37.489359+0000","last_unstale":"2024-04-26T02:24:35.232580+0000","last_undegraded":"2024-04-26T02:24:35.232580+0000","last_fullsized":"2024-04-26T02:24:35.232580+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:47.007579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130163216,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":282,"num_read_kb":274,"num_write":5118,"num_write_kb":178670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"23'3713","reported_seq":3728,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.149169+0000","last_change":"2024-04-26T02:21:37.489974+0000","last_active":"2024-04-26T02:24:35.149169+0000","last_peered":"2024-04-26T02:24:35.149169+0000","last_clean":"2024-04-26T02:24:35.149169+0000","last_became_active":"2024-04-26T02:21:37.488144+0000","last_became_peered":"2024-04-26T02:21:37.488144+0000","last_unstale":"2024-04-26T02:24:35.149169+0000","last_undegraded":"2024-04-26T02:24:35.149169+0000","last_fullsized":"2024-04-26T02:24:35.149169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3713,"log_dups_size":0,"ondisk_log_size":3713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:53:06.375314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":168964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"23'3581","reported_seq":3599,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.139594+0000","last_change":"2024-04-26T02:21:37.495244+0000","last_active":"2024-04-26T02:24:35.139594+0000","last_peered":"2024-04-26T02:24:35.139594+0000","last_clean":"2024-04-26T02:24:35.139594+0000","last_became_active":"2024-04-26T02:21:37.494442+0000","last_became_peered":"2024-04-26T02:21:37.494442+0000","last_unstale":"2024-04-26T02:24:35.139594+0000","last_undegraded":"2024-04-26T02:24:35.139594+0000","last_fullsized":"2024-04-26T02:24:35.139594+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3581,"log_dups_size":0,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:36.918805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":3543,"num_write_kb":168836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"22'4310","reported_seq":4325,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.683294+0000","last_change":"2024-04-26T02:21:37.498351+0000","last_active":"2024-04-26T02:24:34.683294+0000","last_peered":"2024-04-26T02:24:34.683294+0000","last_clean":"2024-04-26T02:24:34.683294+0000","last_became_active":"2024-04-26T02:21:37.497599+0000","last_became_peered":"2024-04-26T02:21:37.497599+0000","last_unstale":"2024-04-26T02:24:34.683294+0000","last_undegraded":"2024-04-26T02:24:34.683294+0000","last_fullsized":"2024-04-26T02:24:34.683294+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:05:22.895436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4304,"num_write_kb":209927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"23'4328","reported_seq":4349,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.137635+0000","last_change":"2024-04-26T02:21:37.487437+0000","last_active":"2024-04-26T02:24:35.137635+0000","last_peered":"2024-04-26T02:24:35.137635+0000","last_clean":"2024-04-26T02:24:35.137635+0000","last_became_active":"2024-04-26T02:21:37.486967+0000","last_became_peered":"2024-04-26T02:21:37.486967+0000","last_unstale":"2024-04-26T02:24:35.137635+0000","last_undegraded":"2024-04-26T02:24:35.137635+0000","last_fullsized":"2024-04-26T02:24:35.137635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:50:14.560813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127401984,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23,"num_read_kb":80,"num_write":4288,"num_write_kb":199362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"23'8884","reported_seq":8927,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:47.788241+0000","last_change":"2024-04-26T02:21:37.496905+0000","last_active":"2024-04-26T02:24:47.788241+0000","last_peered":"2024-04-26T02:24:47.788241+0000","last_clean":"2024-04-26T02:24:47.788241+0000","last_became_active":"2024-04-26T02:21:37.488117+0000","last_became_peered":"2024-04-26T02:21:37.488117+0000","last_unstale":"2024-04-26T02:24:47.788241+0000","last_undegraded":"2024-04-26T02:24:47.788241+0000","last_fullsized":"2024-04-26T02:24:47.788241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":8884,"log_dups_size":0,"ondisk_log_size":8884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:55:56.651035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128582144,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":108,"num_read_kb":91,"num_write":8909,"num_write_kb":228167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3568399112,"num_objects":878,"num_object_clones":0,"num_object_copies":1756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":878,"num_whiteouts":0,"num_read":2493,"num_read_kb":2504,"num_write":159711,"num_write_kb":5636573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6797901824,"data_stored":6835447960,"data_compressed":194293810,"data_compressed_allocated":202072064,"data_compressed_original":243892224,"omap_allocated":0,"internal_metadata":0},"log_size":161261,"ondisk_log_size":161261,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11166,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542199,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1412416,"kb_used_data":1038392,"kb_used_omap":1,"kb_used_meta":374014,"kb_avail":92959424,"statfs":{"total":96636764160,"available":95190450176,"internally_reserved":0,"allocated":1063313408,"data_stored":1066332489,"data_compressed":20319008,"data_compressed_allocated":21159936,"data_compressed_original":25022464,"omap_allocated":1590,"internal_metadata":382990794},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542200,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1148328,"kb_used_data":864544,"kb_used_omap":1,"kb_used_meta":283774,"kb_avail":93223512,"statfs":{"total":96636764160,"available":95460876288,"internally_reserved":0,"allocated":885293056,"data_stored":888264065,"data_compressed":21380520,"data_compressed_allocated":22167552,"data_compressed_original":25878528,"omap_allocated":1595,"internal_metadata":290585029},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542199,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1944992,"kb_used_data":1386328,"kb_used_omap":1,"kb_used_meta":558654,"kb_avail":92426848,"statfs":{"total":96636764160,"available":94645092352,"internally_reserved":0,"allocated":1419599872,"data_stored":1427702961,"data_compressed":38724772,"data_compressed_allocated":40361984,"data_compressed_original":49508352,"omap_allocated":1595,"internal_metadata":572062149},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542201,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1446344,"kb_used_data":1076480,"kb_used_omap":1,"kb_used_meta":369854,"kb_avail":92925496,"statfs":{"total":96636764160,"available":95155707904,"internally_reserved":0,"allocated":1102315520,"data_stored":1108245281,"data_compressed":31089397,"data_compressed_allocated":32296960,"data_compressed_original":39075840,"omap_allocated":1592,"internal_metadata":378730952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542199,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":885764,"kb_used_data":615868,"kb_used_omap":1,"kb_used_meta":269886,"kb_avail":93486076,"statfs":{"total":96636764160,"available":95729741824,"internally_reserved":0,"allocated":630648832,"data_stored":637297197,"data_compressed":27771469,"data_compressed_allocated":28930048,"data_compressed_original":36147200,"omap_allocated":1595,"internal_metadata":276363717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542199,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1387388,"kb_used_data":1018932,"kb_used_omap":1,"kb_used_meta":368446,"kb_avail":92984452,"statfs":{"total":96636764160,"available":95216078848,"internally_reserved":0,"allocated":1043386368,"data_stored":1049606709,"data_compressed":28674963,"data_compressed_allocated":29855744,"data_compressed_original":36777984,"omap_allocated":1595,"internal_metadata":377289157},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464488,"kb_used_data":343072,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93907352,"statfs":{"total":96636764160,"available":96161128448,"internally_reserved":0,"allocated":351305728,"data_stored":352436329,"data_compressed":9222947,"data_compressed_allocated":9637888,"data_compressed_original":11067392,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466656,"kb_used_data":296728,"kb_used_omap":1,"kb_used_meta":169918,"kb_avail":93905184,"statfs":{"total":96636764160,"available":96158908416,"internally_reserved":0,"allocated":303849472,"data_stored":307208889,"data_compressed":17121900,"data_compressed_allocated":17727488,"data_compressed_original":21438464,"omap_allocated":1595,"internal_metadata":173996485},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1043120128,"data_stored":1049004268,"data_compressed":28669380,"data_compressed_allocated":29822976,"data_compressed_original":36265984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":630435840,"data_stored":637223684,"data_compressed":27771469,"data_compressed_allocated":28930048,"data_compressed_original":36147200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303636480,"data_stored":307135376,"data_compressed":17121900,"data_compressed_allocated":17727488,"data_compressed_original":21438464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1102102528,"data_stored":1108171768,"data_compressed":31089397,"data_compressed_allocated":32296960,"data_compressed_original":39075840,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1419386880,"data_stored":1427629448,"data_compressed":38724772,"data_compressed_allocated":40361984,"data_compressed_original":49508352,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":885080064,"data_stored":888190552,"data_compressed":21380520,"data_compressed_allocated":22167552,"data_compressed_original":25878528,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":351092736,"data_stored":352362816,"data_compressed":9222947,"data_compressed_allocated":9637888,"data_compressed_original":11067392,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1063047168,"data_stored":1065730048,"data_compressed":20313425,"data_compressed_allocated":21127168,"data_compressed_original":24510464,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:51.896 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T02:25:51.897 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:52.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda21fe8640 1 Processor -- start 2024-04-26T02:25:52.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda21fe8640 1 -- start start 2024-04-26T02:25:52.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda21fe8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1c0aa220 con 0x7fda1c1402e0 2024-04-26T02:25:52.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda21fe8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1c13f3f0 con 0x7fda1c06f490 2024-04-26T02:25:52.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda21fe8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1c13f5c0 con 0x7fda1c06f0b0 2024-04-26T02:25:52.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda1b7fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fda1c06f0b0 0x7fda1c0a8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38850/0 (socket says 172.21.15.87:38850) 2024-04-26T02:25:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda1b7fe640 1 -- 172.21.15.87:0/4027193111 learned_addr learned my addr 172.21.15.87:0/4027193111 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda1a7fc640 1 -- 172.21.15.87:0/4027193111 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3110344659 0 0) 0x7fda0c001120 con 0x7fda1c06f0b0 2024-04-26T02:25:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda1a7fc640 1 -- 172.21.15.87:0/4027193111 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda08003530 con 0x7fda1c06f0b0 2024-04-26T02:25:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda1a7fc640 1 -- 172.21.15.87:0/4027193111 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1828137325 0 0) 0x7fda1c0aa220 con 0x7fda1c1402e0 2024-04-26T02:25:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda1a7fc640 1 -- 172.21.15.87:0/4027193111 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda0c001120 con 0x7fda1c1402e0 2024-04-26T02:25:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda1a7fc640 1 -- 172.21.15.87:0/4027193111 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 898065916 0 0) 0x7fda08003530 con 0x7fda1c06f0b0 2024-04-26T02:25:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda1a7fc640 1 -- 172.21.15.87:0/4027193111 >> v1:172.21.15.122:6789/0 conn(0x7fda1c06f490 legacy=0x7fda1c13fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda1a7fc640 1 -- 172.21.15.87:0/4027193111 >> v1:172.21.15.87:6789/0 conn(0x7fda1c1402e0 legacy=0x7fda1c142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda1a7fc640 1 -- 172.21.15.87:0/4027193111 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1c143460 con 0x7fda1c06f0b0 2024-04-26T02:25:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda1a7fc640 1 -- 172.21.15.87:0/4027193111 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fda0c0017f0 con 0x7fda1c06f0b0 2024-04-26T02:25:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda1a7fc640 1 -- 172.21.15.87:0/4027193111 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda0c001b10 con 0x7fda1c06f0b0 2024-04-26T02:25:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda1a7fc640 1 -- 172.21.15.87:0/4027193111 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fda0c004a50 con 0x7fda1c06f0b0 2024-04-26T02:25:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda21fe8640 1 -- 172.21.15.87:0/4027193111 >> v1:172.21.15.87:6790/0 conn(0x7fda1c06f0b0 legacy=0x7fda1c0a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda21fe8640 1 -- 172.21.15.87:0/4027193111 shutdown_connections 2024-04-26T02:25:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda21fe8640 1 -- 172.21.15.87:0/4027193111 >> 172.21.15.87:0/4027193111 conn(0x7fda1c0b40d0 msgr2=0x7fda1c0b44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:52.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda21fe8640 1 -- 172.21.15.87:0/4027193111 shutdown_connections 2024-04-26T02:25:52.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda21fe8640 1 -- 172.21.15.87:0/4027193111 wait complete. 2024-04-26T02:25:52.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda21fe8640 1 Processor -- start 2024-04-26T02:25:52.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.012+0000 7fda21fe8640 1 -- start start 2024-04-26T02:25:52.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda21fe8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1c1317a0 con 0x7fda1c1402e0 2024-04-26T02:25:52.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda21fe8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1c131f10 con 0x7fda1c06f490 2024-04-26T02:25:52.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda21fe8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1c13cbf0 con 0x7fda1c06f0b0 2024-04-26T02:25:52.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda1b7fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fda1c06f0b0 0x7fda1c1daae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38856/0 (socket says 172.21.15.87:38856) 2024-04-26T02:25:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda1b7fe640 1 -- 172.21.15.87:0/2386948147 learned_addr learned my addr 172.21.15.87:0/2386948147 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda1bfff640 1 --1- 172.21.15.87:0/2386948147 >> v1:172.21.15.87:6789/0 conn(0x7fda1c1402e0 0x7fda1c12efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51772/0 (socket says 172.21.15.87:51772) 2024-04-26T02:25:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 13763196 0 0) 0x7fda1c13cbf0 con 0x7fda1c06f0b0 2024-04-26T02:25:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9fc003530 con 0x7fda1c06f0b0 2024-04-26T02:25:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3009450312 0 0) 0x7fda1c1317a0 con 0x7fda1c1402e0 2024-04-26T02:25:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda1c13cbf0 con 0x7fda1c1402e0 2024-04-26T02:25:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 848342081 0 0) 0x7fd9fc003530 con 0x7fda1c06f0b0 2024-04-26T02:25:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda1c1317a0 con 0x7fda1c06f0b0 2024-04-26T02:25:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2120534350 0 0) 0x7fda1c13cbf0 con 0x7fda1c1402e0 2024-04-26T02:25:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9fc003530 con 0x7fda1c1402e0 2024-04-26T02:25:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fda0c0017b0 con 0x7fda1c06f0b0 2024-04-26T02:25:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fda14001c00 con 0x7fda1c1402e0 2024-04-26T02:25:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3206015461 0 0) 0x7fda1c131f10 con 0x7fda1c06f490 2024-04-26T02:25:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda1c13cbf0 con 0x7fda1c06f490 2024-04-26T02:25:52.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2256637891 0 0) 0x7fd9fc003530 con 0x7fda1c1402e0 2024-04-26T02:25:52.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 >> v1:172.21.15.87:6790/0 conn(0x7fda1c06f0b0 legacy=0x7fda1c1daae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:52.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 >> v1:172.21.15.122:6789/0 conn(0x7fda1c06f490 legacy=0x7fda1c142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:52.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1c12fcf0 con 0x7fda1c1402e0 2024-04-26T02:25:52.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda21fe8640 1 -- 172.21.15.87:0/2386948147 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fda1c12f6d0 con 0x7fda1c1402e0 2024-04-26T02:25:52.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda21fe8640 1 -- 172.21.15.87:0/2386948147 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fda1c070bf0 con 0x7fda1c1402e0 2024-04-26T02:25:52.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda14003980 con 0x7fda1c1402e0 2024-04-26T02:25:52.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fda14003cd0 con 0x7fda1c1402e0 2024-04-26T02:25:52.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda21fe8640 1 -- 172.21.15.87:0/2386948147 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda00005090 con 0x7fda1c1402e0 2024-04-26T02:25:52.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7fda14003e70 con 0x7fda1c1402e0 2024-04-26T02:25:52.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.016+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7fda140813d0 con 0x7fda1c1402e0 2024-04-26T02:25:52.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.020+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fda140522a0 con 0x7fda1c1402e0 2024-04-26T02:25:52.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.164+0000 7fda21fe8640 1 -- 172.21.15.87:0/2386948147 --> v1:172.21.15.122:6816/416193625 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fda00000ba0 con 0x7fd9fc0683a0 2024-04-26T02:25:52.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.176+0000 7fda13fff640 1 -- 172.21.15.87:0/2386948147 <== mgr.4108 v1:172.21.15.122:6816/416193625 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+117694 (unknown 2965378022 0 2277588350) 0x7fda00000ba0 con 0x7fd9fc0683a0 2024-04-26T02:25:52.178 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:25:52.179 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-26T02:25:52.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.176+0000 7fda21fe8640 1 -- 172.21.15.87:0/2386948147 >> v1:172.21.15.122:6816/416193625 conn(0x7fd9fc0683a0 legacy=0x7fd9fc06a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:52.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.176+0000 7fda21fe8640 1 -- 172.21.15.87:0/2386948147 >> v1:172.21.15.87:6789/0 conn(0x7fda1c1402e0 legacy=0x7fda1c12efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:52.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.176+0000 7fda21fe8640 1 -- 172.21.15.87:0/2386948147 shutdown_connections 2024-04-26T02:25:52.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.176+0000 7fda21fe8640 1 -- 172.21.15.87:0/2386948147 >> 172.21.15.87:0/2386948147 conn(0x7fda1c0b40d0 msgr2=0x7fda1c132710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:52.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.176+0000 7fda21fe8640 1 -- 172.21.15.87:0/2386948147 shutdown_connections 2024-04-26T02:25:52.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.176+0000 7fda21fe8640 1 -- 172.21.15.87:0/2386948147 wait complete. 2024-04-26T02:25:52.194 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-04-26T02:25:51.415976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3568989480,"num_objects":880,"num_object_clones":0,"num_object_copies":1760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":2609,"num_read_kb":2604,"num_write":159870,"num_write_kb":5638989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161378,"ondisk_log_size":161378,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":9156376,"kb_used_data":6640344,"kb_used_omap":12,"kb_used_meta":2515955,"kb_avail":745818344,"statfs":{"total":773094113280,"available":763717984256,"internally_reserved":0,"allocated":6799712256,"data_stored":6837093920,"data_compressed":194304976,"data_compressed_allocated":202137600,"data_compressed_original":244916224,"omap_allocated":12751,"internal_metadata":2576338481},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36864,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003159"},"pg_stats":[{"pgid":"3.1f","version":"22'2749","reported_seq":2768,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.658393+0000","last_change":"2024-04-26T02:21:37.492661+0000","last_active":"2024-04-26T02:24:34.658393+0000","last_peered":"2024-04-26T02:24:34.658393+0000","last_clean":"2024-04-26T02:24:34.658393+0000","last_became_active":"2024-04-26T02:21:37.491078+0000","last_became_peered":"2024-04-26T02:21:37.491078+0000","last_unstale":"2024-04-26T02:24:34.658393+0000","last_undegraded":"2024-04-26T02:24:34.658393+0000","last_fullsized":"2024-04-26T02:24:34.658393+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":2749,"log_dups_size":0,"ondisk_log_size":2749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:56:51.567611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":71,"num_write":2855,"num_write_kb":123586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"23'6576","reported_seq":6592,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:42.266530+0000","last_change":"2024-04-26T02:21:37.485093+0000","last_active":"2024-04-26T02:24:42.266530+0000","last_peered":"2024-04-26T02:24:42.266530+0000","last_clean":"2024-04-26T02:24:42.266530+0000","last_became_active":"2024-04-26T02:21:37.484287+0000","last_became_peered":"2024-04-26T02:21:37.484287+0000","last_unstale":"2024-04-26T02:24:42.266530+0000","last_undegraded":"2024-04-26T02:24:42.266530+0000","last_fullsized":"2024-04-26T02:24:42.266530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":6576,"log_dups_size":0,"ondisk_log_size":6576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:56:08.755136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6549,"num_write_kb":206212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"23'5724","reported_seq":5759,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:47.550994+0000","last_change":"2024-04-26T02:21:37.498519+0000","last_active":"2024-04-26T02:24:47.550994+0000","last_peered":"2024-04-26T02:24:47.550994+0000","last_clean":"2024-04-26T02:24:47.550994+0000","last_became_active":"2024-04-26T02:21:37.497700+0000","last_became_peered":"2024-04-26T02:21:37.497700+0000","last_unstale":"2024-04-26T02:24:47.550994+0000","last_undegraded":"2024-04-26T02:24:47.550994+0000","last_fullsized":"2024-04-26T02:24:47.550994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":5724,"log_dups_size":0,"ondisk_log_size":5724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:02:58.912915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":76,"num_read_kb":124,"num_write":5736,"num_write_kb":155565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"22'4059","reported_seq":4077,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.734233+0000","last_change":"2024-04-26T02:21:37.497650+0000","last_active":"2024-04-26T02:24:34.734233+0000","last_peered":"2024-04-26T02:24:34.734233+0000","last_clean":"2024-04-26T02:24:34.734233+0000","last_became_active":"2024-04-26T02:21:37.496863+0000","last_became_peered":"2024-04-26T02:21:37.496863+0000","last_unstale":"2024-04-26T02:24:34.734233+0000","last_undegraded":"2024-04-26T02:24:34.734233+0000","last_fullsized":"2024-04-26T02:24:34.734233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4059,"log_dups_size":0,"ondisk_log_size":4059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:57:07.038484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":4059,"num_write_kb":185415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"23'3948","reported_seq":3972,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.141201+0000","last_change":"2024-04-26T02:21:37.491890+0000","last_active":"2024-04-26T02:24:35.141201+0000","last_peered":"2024-04-26T02:24:35.141201+0000","last_clean":"2024-04-26T02:24:35.141201+0000","last_became_active":"2024-04-26T02:21:37.490474+0000","last_became_peered":"2024-04-26T02:21:37.490474+0000","last_unstale":"2024-04-26T02:24:35.141201+0000","last_undegraded":"2024-04-26T02:24:35.141201+0000","last_fullsized":"2024-04-26T02:24:35.141201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3948,"log_dups_size":0,"ondisk_log_size":3948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:56.993069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120324096,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":34,"num_read_kb":151,"num_write":3926,"num_write_kb":190983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"23'6081","reported_seq":6102,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:41.034089+0000","last_change":"2024-04-26T02:22:35.114133+0000","last_active":"2024-04-26T02:24:41.034089+0000","last_peered":"2024-04-26T02:24:41.034089+0000","last_clean":"2024-04-26T02:24:41.034089+0000","last_became_active":"2024-04-26T02:21:37.485110+0000","last_became_peered":"2024-04-26T02:21:37.485110+0000","last_unstale":"2024-04-26T02:24:41.034089+0000","last_undegraded":"2024-04-26T02:24:41.034089+0000","last_fullsized":"2024-04-26T02:24:41.034089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":6081,"log_dups_size":0,"ondisk_log_size":6081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:39:18.389527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":6064,"num_write_kb":187005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"23'5039","reported_seq":5073,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:52.103217+0000","last_change":"2024-04-26T02:21:37.489148+0000","last_active":"2024-04-26T02:24:52.103217+0000","last_peered":"2024-04-26T02:24:52.103217+0000","last_clean":"2024-04-26T02:24:52.103217+0000","last_became_active":"2024-04-26T02:21:37.488519+0000","last_became_peered":"2024-04-26T02:21:37.488519+0000","last_unstale":"2024-04-26T02:24:52.103217+0000","last_undegraded":"2024-04-26T02:24:52.103217+0000","last_fullsized":"2024-04-26T02:24:52.103217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:41.178413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117441024,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":5034,"num_write_kb":181716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"23'7153","reported_seq":7190,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:52.126229+0000","last_change":"2024-04-26T02:21:37.496383+0000","last_active":"2024-04-26T02:24:52.126229+0000","last_peered":"2024-04-26T02:24:52.126229+0000","last_clean":"2024-04-26T02:24:52.126229+0000","last_became_active":"2024-04-26T02:21:37.495694+0000","last_became_peered":"2024-04-26T02:21:37.495694+0000","last_unstale":"2024-04-26T02:24:52.126229+0000","last_undegraded":"2024-04-26T02:24:52.126229+0000","last_fullsized":"2024-04-26T02:24:52.126229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":7153,"log_dups_size":0,"ondisk_log_size":7153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:27:30.478083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023936,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":80,"num_read_kb":65,"num_write":7184,"num_write_kb":197893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"23'8281","reported_seq":8304,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:25:39.938993+0000","last_change":"2024-04-26T02:21:37.490841+0000","last_active":"2024-04-26T02:25:39.938993+0000","last_peered":"2024-04-26T02:25:39.938993+0000","last_clean":"2024-04-26T02:25:39.938993+0000","last_became_active":"2024-04-26T02:21:37.490119+0000","last_became_peered":"2024-04-26T02:21:37.490119+0000","last_unstale":"2024-04-26T02:25:39.938993+0000","last_undegraded":"2024-04-26T02:25:39.938993+0000","last_fullsized":"2024-04-26T02:25:39.938993+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":8281,"log_dups_size":0,"ondisk_log_size":8281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:36:46.649805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137060352,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":8291,"num_write_kb":229215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"23'3756","reported_seq":3771,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.020013+0000","last_change":"2024-04-26T02:21:37.507074+0000","last_active":"2024-04-26T02:24:35.020013+0000","last_peered":"2024-04-26T02:24:35.020013+0000","last_clean":"2024-04-26T02:24:35.020013+0000","last_became_active":"2024-04-26T02:21:37.506309+0000","last_became_peered":"2024-04-26T02:21:37.506309+0000","last_unstale":"2024-04-26T02:24:35.020013+0000","last_undegraded":"2024-04-26T02:24:35.020013+0000","last_fullsized":"2024-04-26T02:24:35.020013+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3756,"log_dups_size":0,"ondisk_log_size":3756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:49:23.247821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3724,"num_write_kb":119684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"23'7177","reported_seq":7192,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:50.387812+0000","last_change":"2024-04-26T02:21:37.487697+0000","last_active":"2024-04-26T02:24:50.387812+0000","last_peered":"2024-04-26T02:24:50.387812+0000","last_clean":"2024-04-26T02:24:50.387812+0000","last_became_active":"2024-04-26T02:21:37.486868+0000","last_became_peered":"2024-04-26T02:21:37.486868+0000","last_unstale":"2024-04-26T02:24:50.387812+0000","last_undegraded":"2024-04-26T02:24:50.387812+0000","last_fullsized":"2024-04-26T02:24:50.387812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":7177,"log_dups_size":0,"ondisk_log_size":7177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:26:13.460721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7176,"num_write_kb":198373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"23'4534","reported_seq":4549,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:44.307401+0000","last_change":"2024-04-26T02:21:37.488921+0000","last_active":"2024-04-26T02:24:44.307401+0000","last_peered":"2024-04-26T02:24:44.307401+0000","last_clean":"2024-04-26T02:24:44.307401+0000","last_became_active":"2024-04-26T02:21:37.487980+0000","last_became_peered":"2024-04-26T02:21:37.487980+0000","last_unstale":"2024-04-26T02:24:44.307401+0000","last_undegraded":"2024-04-26T02:24:44.307401+0000","last_fullsized":"2024-04-26T02:24:44.307401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4534,"log_dups_size":0,"ondisk_log_size":4534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:10.065671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4488,"num_write_kb":107395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.730342+0000","last_change":"2024-04-26T02:21:31.429340+0000","last_active":"2024-04-26T02:24:34.730342+0000","last_peered":"2024-04-26T02:24:34.730342+0000","last_clean":"2024-04-26T02:24:34.730342+0000","last_became_active":"2024-04-26T02:21:31.428657+0000","last_became_peered":"2024-04-26T02:21:31.428657+0000","last_unstale":"2024-04-26T02:24:34.730342+0000","last_undegraded":"2024-04-26T02:24:34.730342+0000","last_fullsized":"2024-04-26T02:24:34.730342+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:07:12.674824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'4886","reported_seq":4909,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:48.178325+0000","last_change":"2024-04-26T02:21:37.482817+0000","last_active":"2024-04-26T02:24:48.178325+0000","last_peered":"2024-04-26T02:24:48.178325+0000","last_clean":"2024-04-26T02:24:48.178325+0000","last_became_active":"2024-04-26T02:21:37.481620+0000","last_became_peered":"2024-04-26T02:21:37.481620+0000","last_unstale":"2024-04-26T02:24:48.178325+0000","last_undegraded":"2024-04-26T02:24:48.178325+0000","last_fullsized":"2024-04-26T02:24:48.178325+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4886,"log_dups_size":0,"ondisk_log_size":4886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:26.760929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96600064,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":4875,"num_write_kb":131621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.658277+0000","last_change":"2024-04-26T02:21:31.432330+0000","last_active":"2024-04-26T02:24:34.658277+0000","last_peered":"2024-04-26T02:24:34.658277+0000","last_clean":"2024-04-26T02:24:34.658277+0000","last_became_active":"2024-04-26T02:21:31.431958+0000","last_became_peered":"2024-04-26T02:21:31.431958+0000","last_unstale":"2024-04-26T02:24:34.658277+0000","last_undegraded":"2024-04-26T02:24:34.658277+0000","last_fullsized":"2024-04-26T02:24:34.658277+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:59:02.906743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'4222","reported_seq":4237,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.143067+0000","last_change":"2024-04-26T02:21:37.488836+0000","last_active":"2024-04-26T02:24:35.143067+0000","last_peered":"2024-04-26T02:24:35.143067+0000","last_clean":"2024-04-26T02:24:35.143067+0000","last_became_active":"2024-04-26T02:21:37.487980+0000","last_became_peered":"2024-04-26T02:21:37.487980+0000","last_unstale":"2024-04-26T02:24:35.143067+0000","last_undegraded":"2024-04-26T02:24:35.143067+0000","last_fullsized":"2024-04-26T02:24:35.143067+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:07:49.262710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4175,"num_write_kb":201729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.684259+0000","last_change":"2024-04-26T02:21:31.430988+0000","last_active":"2024-04-26T02:24:34.684259+0000","last_peered":"2024-04-26T02:24:34.684259+0000","last_clean":"2024-04-26T02:24:34.684259+0000","last_became_active":"2024-04-26T02:21:31.430227+0000","last_became_peered":"2024-04-26T02:21:31.430227+0000","last_unstale":"2024-04-26T02:24:34.684259+0000","last_undegraded":"2024-04-26T02:24:34.684259+0000","last_fullsized":"2024-04-26T02:24:34.684259+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:32:21.265195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'6121","reported_seq":6142,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:54.144536+0000","last_change":"2024-04-26T02:21:37.483474+0000","last_active":"2024-04-26T02:24:54.144536+0000","last_peered":"2024-04-26T02:24:54.144536+0000","last_clean":"2024-04-26T02:24:54.144536+0000","last_became_active":"2024-04-26T02:21:37.482894+0000","last_became_peered":"2024-04-26T02:21:37.482894+0000","last_unstale":"2024-04-26T02:24:54.144536+0000","last_undegraded":"2024-04-26T02:24:54.144536+0000","last_fullsized":"2024-04-26T02:24:54.144536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":6121,"log_dups_size":0,"ondisk_log_size":6121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:14.817911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99354976,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":6134,"num_write_kb":151141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.659060+0000","last_change":"2024-04-26T02:21:31.431358+0000","last_active":"2024-04-26T02:24:34.659060+0000","last_peered":"2024-04-26T02:24:34.659060+0000","last_clean":"2024-04-26T02:24:34.659060+0000","last_became_active":"2024-04-26T02:21:31.431071+0000","last_became_peered":"2024-04-26T02:21:31.431071+0000","last_unstale":"2024-04-26T02:24:34.659060+0000","last_undegraded":"2024-04-26T02:24:34.659060+0000","last_fullsized":"2024-04-26T02:24:34.659060+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:41:07.918405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"22'4180","reported_seq":4225,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.735876+0000","last_change":"2024-04-26T02:21:37.490970+0000","last_active":"2024-04-26T02:24:34.735876+0000","last_peered":"2024-04-26T02:24:34.735876+0000","last_clean":"2024-04-26T02:24:34.735876+0000","last_became_active":"2024-04-26T02:21:37.490105+0000","last_became_peered":"2024-04-26T02:21:37.490105+0000","last_unstale":"2024-04-26T02:24:34.735876+0000","last_undegraded":"2024-04-26T02:24:34.735876+0000","last_fullsized":"2024-04-26T02:24:34.735876+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4180,"log_dups_size":0,"ondisk_log_size":4180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:17.335251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121896960,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":122,"num_read_kb":98,"num_write":4227,"num_write_kb":167240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"22'2962","reported_seq":2980,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.670117+0000","last_change":"2024-04-26T02:21:37.494426+0000","last_active":"2024-04-26T02:24:34.670117+0000","last_peered":"2024-04-26T02:24:34.670117+0000","last_clean":"2024-04-26T02:24:34.670117+0000","last_became_active":"2024-04-26T02:21:37.491249+0000","last_became_peered":"2024-04-26T02:21:37.491249+0000","last_unstale":"2024-04-26T02:24:34.670117+0000","last_undegraded":"2024-04-26T02:24:34.670117+0000","last_fullsized":"2024-04-26T02:24:34.670117+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:22:39.029300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74842112,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":7,"num_write":2966,"num_write_kb":163586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.734132+0000","last_change":"2024-04-26T02:21:32.425721+0000","last_active":"2024-04-26T02:24:34.734132+0000","last_peered":"2024-04-26T02:24:34.734132+0000","last_clean":"2024-04-26T02:24:34.734132+0000","last_became_active":"2024-04-26T02:21:32.425189+0000","last_became_peered":"2024-04-26T02:21:32.425189+0000","last_unstale":"2024-04-26T02:24:34.734132+0000","last_undegraded":"2024-04-26T02:24:34.734132+0000","last_fullsized":"2024-04-26T02:24:34.734132+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:56:47.156837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"23'3481","reported_seq":3503,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.146368+0000","last_change":"2024-04-26T02:21:37.486201+0000","last_active":"2024-04-26T02:24:35.146368+0000","last_peered":"2024-04-26T02:24:35.146368+0000","last_clean":"2024-04-26T02:24:35.146368+0000","last_became_active":"2024-04-26T02:21:37.485432+0000","last_became_peered":"2024-04-26T02:21:37.485432+0000","last_unstale":"2024-04-26T02:24:35.146368+0000","last_undegraded":"2024-04-26T02:24:35.146368+0000","last_fullsized":"2024-04-26T02:24:35.146368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:16:36.622533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86706520,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":3452,"num_write_kb":123177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.676622+0000","last_change":"2024-04-26T02:21:31.436585+0000","last_active":"2024-04-26T02:24:34.676622+0000","last_peered":"2024-04-26T02:24:34.676622+0000","last_clean":"2024-04-26T02:24:34.676622+0000","last_became_active":"2024-04-26T02:21:31.436226+0000","last_became_peered":"2024-04-26T02:21:31.436226+0000","last_unstale":"2024-04-26T02:24:34.676622+0000","last_undegraded":"2024-04-26T02:24:34.676622+0000","last_fullsized":"2024-04-26T02:24:34.676622+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:21:29.470825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"23'4455","reported_seq":4472,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.147147+0000","last_change":"2024-04-26T02:21:37.494384+0000","last_active":"2024-04-26T02:24:35.147147+0000","last_peered":"2024-04-26T02:24:35.147147+0000","last_clean":"2024-04-26T02:24:35.147147+0000","last_became_active":"2024-04-26T02:21:37.493922+0000","last_became_peered":"2024-04-26T02:21:37.493922+0000","last_unstale":"2024-04-26T02:24:35.147147+0000","last_undegraded":"2024-04-26T02:24:35.147147+0000","last_fullsized":"2024-04-26T02:24:35.147147+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4455,"log_dups_size":0,"ondisk_log_size":4455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:42:04.618288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113312264,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":4436,"num_write_kb":167962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.683586+0000","last_change":"2024-04-26T02:21:31.433313+0000","last_active":"2024-04-26T02:24:34.683586+0000","last_peered":"2024-04-26T02:24:34.683586+0000","last_clean":"2024-04-26T02:24:34.683586+0000","last_became_active":"2024-04-26T02:21:31.432395+0000","last_became_peered":"2024-04-26T02:21:31.432395+0000","last_unstale":"2024-04-26T02:24:34.683586+0000","last_undegraded":"2024-04-26T02:24:34.683586+0000","last_fullsized":"2024-04-26T02:24:34.683586+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:57:44.139176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"23'4930","reported_seq":4959,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:49.161666+0000","last_change":"2024-04-26T02:21:37.485786+0000","last_active":"2024-04-26T02:24:49.161666+0000","last_peered":"2024-04-26T02:24:49.161666+0000","last_clean":"2024-04-26T02:24:49.161666+0000","last_became_active":"2024-04-26T02:21:37.484172+0000","last_became_peered":"2024-04-26T02:21:37.484172+0000","last_unstale":"2024-04-26T02:24:49.161666+0000","last_undegraded":"2024-04-26T02:24:49.161666+0000","last_fullsized":"2024-04-26T02:24:49.161666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:40:57.948908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91030032,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":49,"num_read_kb":40,"num_write":4960,"num_write_kb":192278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.734226+0000","last_change":"2024-04-26T02:21:33.341715+0000","last_active":"2024-04-26T02:24:34.734226+0000","last_peered":"2024-04-26T02:24:34.734226+0000","last_clean":"2024-04-26T02:24:34.734226+0000","last_became_active":"2024-04-26T02:21:33.340937+0000","last_became_peered":"2024-04-26T02:21:33.340937+0000","last_unstale":"2024-04-26T02:24:34.734226+0000","last_undegraded":"2024-04-26T02:24:34.734226+0000","last_fullsized":"2024-04-26T02:24:34.734226+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:03:17.530982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.683452+0000","last_change":"2024-04-26T02:21:29.434185+0000","last_active":"2024-04-26T02:24:34.683452+0000","last_peered":"2024-04-26T02:24:34.683452+0000","last_clean":"2024-04-26T02:24:34.683452+0000","last_became_active":"2024-04-26T02:21:29.432363+0000","last_became_peered":"2024-04-26T02:21:29.432363+0000","last_unstale":"2024-04-26T02:24:34.683452+0000","last_undegraded":"2024-04-26T02:24:34.683452+0000","last_fullsized":"2024-04-26T02:24:34.683452+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:28.375870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:28.375870+0000","last_clean_scrub_stamp":"2024-04-26T02:21:28.375870+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:18:11.058445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"23'4891","reported_seq":4906,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.147573+0000","last_change":"2024-04-26T02:21:37.488171+0000","last_active":"2024-04-26T02:24:35.147573+0000","last_peered":"2024-04-26T02:24:35.147573+0000","last_clean":"2024-04-26T02:24:35.147573+0000","last_became_active":"2024-04-26T02:21:37.486673+0000","last_became_peered":"2024-04-26T02:21:37.486673+0000","last_unstale":"2024-04-26T02:24:35.147573+0000","last_undegraded":"2024-04-26T02:24:35.147573+0000","last_fullsized":"2024-04-26T02:24:35.147573+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4891,"log_dups_size":0,"ondisk_log_size":4891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:29:05.728663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4819,"num_write_kb":226567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"23'9645","reported_seq":9761,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:51.709831+0000","last_change":"2024-04-26T02:21:37.476830+0000","last_active":"2024-04-26T02:24:51.709831+0000","last_peered":"2024-04-26T02:24:51.709831+0000","last_clean":"2024-04-26T02:24:51.709831+0000","last_became_active":"2024-04-26T02:21:37.476072+0000","last_became_peered":"2024-04-26T02:21:37.476072+0000","last_unstale":"2024-04-26T02:24:51.709831+0000","last_undegraded":"2024-04-26T02:24:51.709831+0000","last_fullsized":"2024-04-26T02:24:51.709831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":9645,"log_dups_size":0,"ondisk_log_size":9645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:26:14.037852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138481144,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":348,"num_read_kb":309,"num_write":7857,"num_write_kb":214099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"23'5768","reported_seq":5892,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:53.974613+0000","last_change":"2024-04-26T02:21:37.490171+0000","last_active":"2024-04-26T02:24:53.974613+0000","last_peered":"2024-04-26T02:24:53.974613+0000","last_clean":"2024-04-26T02:24:53.974613+0000","last_became_active":"2024-04-26T02:21:37.488724+0000","last_became_peered":"2024-04-26T02:21:37.488724+0000","last_unstale":"2024-04-26T02:24:53.974613+0000","last_undegraded":"2024-04-26T02:24:53.974613+0000","last_fullsized":"2024-04-26T02:24:53.974613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":5768,"log_dups_size":0,"ondisk_log_size":5768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:17.512625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134480400,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":132,"num_read_kb":124,"num_write":5774,"num_write_kb":201317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"22'3903","reported_seq":3918,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.656121+0000","last_change":"2024-04-26T02:21:37.492466+0000","last_active":"2024-04-26T02:24:34.656121+0000","last_peered":"2024-04-26T02:24:34.656121+0000","last_clean":"2024-04-26T02:24:34.656121+0000","last_became_active":"2024-04-26T02:21:37.491785+0000","last_became_peered":"2024-04-26T02:21:37.491785+0000","last_unstale":"2024-04-26T02:24:34.656121+0000","last_undegraded":"2024-04-26T02:24:34.656121+0000","last_fullsized":"2024-04-26T02:24:34.656121+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3903,"log_dups_size":0,"ondisk_log_size":3903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:44.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88842240,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3901,"num_write_kb":140906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"23'3918","reported_seq":3935,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.141267+0000","last_change":"2024-04-26T02:21:37.493141+0000","last_active":"2024-04-26T02:24:35.141267+0000","last_peered":"2024-04-26T02:24:35.141267+0000","last_clean":"2024-04-26T02:24:35.141267+0000","last_became_active":"2024-04-26T02:21:37.490836+0000","last_became_peered":"2024-04-26T02:21:37.490836+0000","last_unstale":"2024-04-26T02:24:35.141267+0000","last_undegraded":"2024-04-26T02:24:35.141267+0000","last_fullsized":"2024-04-26T02:24:35.141267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:35.359899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100672016,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3880,"num_write_kb":189331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"23'3148","reported_seq":3471,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.011294+0000","last_change":"2024-04-26T02:21:37.501742+0000","last_active":"2024-04-26T02:24:35.011294+0000","last_peered":"2024-04-26T02:24:35.011294+0000","last_clean":"2024-04-26T02:24:35.011294+0000","last_became_active":"2024-04-26T02:21:37.500919+0000","last_became_peered":"2024-04-26T02:21:37.500919+0000","last_unstale":"2024-04-26T02:24:35.011294+0000","last_undegraded":"2024-04-26T02:24:35.011294+0000","last_fullsized":"2024-04-26T02:24:35.011294+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:15:44.491969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84148752,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1028,"num_read_kb":921,"num_write":3342,"num_write_kb":128651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"23'4858","reported_seq":4947,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.232580+0000","last_change":"2024-04-26T02:21:37.497393+0000","last_active":"2024-04-26T02:24:35.232580+0000","last_peered":"2024-04-26T02:24:35.232580+0000","last_clean":"2024-04-26T02:24:35.232580+0000","last_became_active":"2024-04-26T02:21:37.489359+0000","last_became_peered":"2024-04-26T02:21:37.489359+0000","last_unstale":"2024-04-26T02:24:35.232580+0000","last_undegraded":"2024-04-26T02:24:35.232580+0000","last_fullsized":"2024-04-26T02:24:35.232580+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:47.007579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130163216,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":282,"num_read_kb":274,"num_write":5118,"num_write_kb":178670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"23'3713","reported_seq":3728,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.149169+0000","last_change":"2024-04-26T02:21:37.489974+0000","last_active":"2024-04-26T02:24:35.149169+0000","last_peered":"2024-04-26T02:24:35.149169+0000","last_clean":"2024-04-26T02:24:35.149169+0000","last_became_active":"2024-04-26T02:21:37.488144+0000","last_became_peered":"2024-04-26T02:21:37.488144+0000","last_unstale":"2024-04-26T02:24:35.149169+0000","last_undegraded":"2024-04-26T02:24:35.149169+0000","last_fullsized":"2024-04-26T02:24:35.149169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3713,"log_dups_size":0,"ondisk_log_size":3713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:53:06.375314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":168964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"23'3581","reported_seq":3599,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.139594+0000","last_change":"2024-04-26T02:21:37.495244+0000","last_active":"2024-04-26T02:24:35.139594+0000","last_peered":"2024-04-26T02:24:35.139594+0000","last_clean":"2024-04-26T02:24:35.139594+0000","last_became_active":"2024-04-26T02:21:37.494442+0000","last_became_peered":"2024-04-26T02:21:37.494442+0000","last_unstale":"2024-04-26T02:24:35.139594+0000","last_undegraded":"2024-04-26T02:24:35.139594+0000","last_fullsized":"2024-04-26T02:24:35.139594+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3581,"log_dups_size":0,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:36.918805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":3543,"num_write_kb":168836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"22'4310","reported_seq":4325,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:34.683294+0000","last_change":"2024-04-26T02:21:37.498351+0000","last_active":"2024-04-26T02:24:34.683294+0000","last_peered":"2024-04-26T02:24:34.683294+0000","last_clean":"2024-04-26T02:24:34.683294+0000","last_became_active":"2024-04-26T02:21:37.497599+0000","last_became_peered":"2024-04-26T02:21:37.497599+0000","last_unstale":"2024-04-26T02:24:34.683294+0000","last_undegraded":"2024-04-26T02:24:34.683294+0000","last_fullsized":"2024-04-26T02:24:34.683294+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:05:22.895436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4304,"num_write_kb":209927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"23'4328","reported_seq":4349,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:35.137635+0000","last_change":"2024-04-26T02:21:37.487437+0000","last_active":"2024-04-26T02:24:35.137635+0000","last_peered":"2024-04-26T02:24:35.137635+0000","last_clean":"2024-04-26T02:24:35.137635+0000","last_became_active":"2024-04-26T02:21:37.486967+0000","last_became_peered":"2024-04-26T02:21:37.486967+0000","last_unstale":"2024-04-26T02:24:35.137635+0000","last_undegraded":"2024-04-26T02:24:35.137635+0000","last_fullsized":"2024-04-26T02:24:35.137635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:50:14.560813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127401984,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23,"num_read_kb":80,"num_write":4288,"num_write_kb":199362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"23'8884","reported_seq":8927,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-26T02:24:47.788241+0000","last_change":"2024-04-26T02:21:37.496905+0000","last_active":"2024-04-26T02:24:47.788241+0000","last_peered":"2024-04-26T02:24:47.788241+0000","last_clean":"2024-04-26T02:24:47.788241+0000","last_became_active":"2024-04-26T02:21:37.488117+0000","last_became_peered":"2024-04-26T02:21:37.488117+0000","last_unstale":"2024-04-26T02:24:47.788241+0000","last_undegraded":"2024-04-26T02:24:47.788241+0000","last_fullsized":"2024-04-26T02:24:47.788241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":8884,"log_dups_size":0,"ondisk_log_size":8884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:55:56.651035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128582144,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":108,"num_read_kb":91,"num_write":8909,"num_write_kb":228167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3568399112,"num_objects":878,"num_object_clones":0,"num_object_copies":1756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":878,"num_whiteouts":0,"num_read":2493,"num_read_kb":2504,"num_write":159711,"num_write_kb":5636573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6797901824,"data_stored":6835447960,"data_compressed":194293810,"data_compressed_allocated":202072064,"data_compressed_original":243892224,"omap_allocated":0,"internal_metadata":0},"log_size":161261,"ondisk_log_size":161261,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11166,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542199,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1412416,"kb_used_data":1038392,"kb_used_omap":1,"kb_used_meta":374014,"kb_avail":92959424,"statfs":{"total":96636764160,"available":95190450176,"internally_reserved":0,"allocated":1063313408,"data_stored":1066332489,"data_compressed":20319008,"data_compressed_allocated":21159936,"data_compressed_original":25022464,"omap_allocated":1590,"internal_metadata":382990794},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542200,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1148328,"kb_used_data":864544,"kb_used_omap":1,"kb_used_meta":283774,"kb_avail":93223512,"statfs":{"total":96636764160,"available":95460876288,"internally_reserved":0,"allocated":885293056,"data_stored":888264065,"data_compressed":21380520,"data_compressed_allocated":22167552,"data_compressed_original":25878528,"omap_allocated":1595,"internal_metadata":290585029},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542199,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1944992,"kb_used_data":1386328,"kb_used_omap":1,"kb_used_meta":558654,"kb_avail":92426848,"statfs":{"total":96636764160,"available":94645092352,"internally_reserved":0,"allocated":1419599872,"data_stored":1427702961,"data_compressed":38724772,"data_compressed_allocated":40361984,"data_compressed_original":49508352,"omap_allocated":1595,"internal_metadata":572062149},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542201,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1446344,"kb_used_data":1076480,"kb_used_omap":1,"kb_used_meta":369854,"kb_avail":92925496,"statfs":{"total":96636764160,"available":95155707904,"internally_reserved":0,"allocated":1102315520,"data_stored":1108245281,"data_compressed":31089397,"data_compressed_allocated":32296960,"data_compressed_original":39075840,"omap_allocated":1592,"internal_metadata":378730952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542199,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":885764,"kb_used_data":615868,"kb_used_omap":1,"kb_used_meta":269886,"kb_avail":93486076,"statfs":{"total":96636764160,"available":95729741824,"internally_reserved":0,"allocated":630648832,"data_stored":637297197,"data_compressed":27771469,"data_compressed_allocated":28930048,"data_compressed_original":36147200,"omap_allocated":1595,"internal_metadata":276363717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542199,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1387388,"kb_used_data":1018932,"kb_used_omap":1,"kb_used_meta":368446,"kb_avail":92984452,"statfs":{"total":96636764160,"available":95216078848,"internally_reserved":0,"allocated":1043386368,"data_stored":1049606709,"data_compressed":28674963,"data_compressed_allocated":29855744,"data_compressed_original":36777984,"omap_allocated":1595,"internal_metadata":377289157},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464488,"kb_used_data":343072,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93907352,"statfs":{"total":96636764160,"available":96161128448,"internally_reserved":0,"allocated":351305728,"data_stored":352436329,"data_compressed":9222947,"data_compressed_allocated":9637888,"data_compressed_original":11067392,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466656,"kb_used_data":296728,"kb_used_omap":1,"kb_used_meta":169918,"kb_avail":93905184,"statfs":{"total":96636764160,"available":96158908416,"internally_reserved":0,"allocated":303849472,"data_stored":307208889,"data_compressed":17121900,"data_compressed_allocated":17727488,"data_compressed_original":21438464,"omap_allocated":1595,"internal_metadata":173996485},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1043120128,"data_stored":1049004268,"data_compressed":28669380,"data_compressed_allocated":29822976,"data_compressed_original":36265984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":630435840,"data_stored":637223684,"data_compressed":27771469,"data_compressed_allocated":28930048,"data_compressed_original":36147200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303636480,"data_stored":307135376,"data_compressed":17121900,"data_compressed_allocated":17727488,"data_compressed_original":21438464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1102102528,"data_stored":1108171768,"data_compressed":31089397,"data_compressed_allocated":32296960,"data_compressed_original":39075840,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1419386880,"data_stored":1427629448,"data_compressed":38724772,"data_compressed_allocated":40361984,"data_compressed_original":49508352,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":885080064,"data_stored":888190552,"data_compressed":21380520,"data_compressed_allocated":22167552,"data_compressed_original":25878528,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":351092736,"data_stored":352362816,"data_compressed":9222947,"data_compressed_allocated":9637888,"data_compressed_original":11067392,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1063047168,"data_stored":1065730048,"data_compressed":20313425,"data_compressed_allocated":21127168,"data_compressed_original":24510464,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:52.197 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T02:25:52.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f7658d23640 1 Processor -- start 2024-04-26T02:25:52.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f7658d23640 1 -- start start 2024-04-26T02:25:52.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f7658d23640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7654071d00 con 0x7f76540b0580 2024-04-26T02:25:52.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f7658d23640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f765406f7c0 con 0x7f7654145c80 2024-04-26T02:25:52.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f7658d23640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7654142420 con 0x7f7654071920 2024-04-26T02:25:52.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f765259c640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f7654071920 0x7f7654146630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38864/0 (socket says 172.21.15.87:38864) 2024-04-26T02:25:52.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f7652d9d640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f76540b0580 0x7f7654141d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51780/0 (socket says 172.21.15.87:51780) 2024-04-26T02:25:52.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f765259c640 1 -- 172.21.15.87:0/2467055798 learned_addr learned my addr 172.21.15.87:0/2467055798 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:52.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f765159a640 1 -- 172.21.15.87:0/2467055798 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 314190655 0 0) 0x7f7654142420 con 0x7f7654071920 2024-04-26T02:25:52.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f765159a640 1 -- 172.21.15.87:0/2467055798 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f763c003530 con 0x7f7654071920 2024-04-26T02:25:52.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f765159a640 1 -- 172.21.15.87:0/2467055798 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1962977145 0 0) 0x7f764c0014a0 con 0x7f76540b0580 2024-04-26T02:25:52.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f765159a640 1 -- 172.21.15.87:0/2467055798 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7654142420 con 0x7f76540b0580 2024-04-26T02:25:52.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f765159a640 1 -- 172.21.15.87:0/2467055798 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1959562905 0 0) 0x7f763c003530 con 0x7f7654071920 2024-04-26T02:25:52.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f765159a640 1 -- 172.21.15.87:0/2467055798 >> v1:172.21.15.122:6789/0 conn(0x7f7654145c80 legacy=0x7f765406f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:52.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f765159a640 1 -- 172.21.15.87:0/2467055798 >> v1:172.21.15.87:6789/0 conn(0x7f76540b0580 legacy=0x7f7654141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:52.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f765159a640 1 -- 172.21.15.87:0/2467055798 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7654072e20 con 0x7f7654071920 2024-04-26T02:25:52.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f765159a640 1 -- 172.21.15.87:0/2467055798 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f7648001620 con 0x7f7654071920 2024-04-26T02:25:52.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f765159a640 1 -- 172.21.15.87:0/2467055798 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7648001900 con 0x7f7654071920 2024-04-26T02:25:52.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f765159a640 1 -- 172.21.15.87:0/2467055798 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f7648001d10 con 0x7f7654071920 2024-04-26T02:25:52.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f7658d23640 1 -- 172.21.15.87:0/2467055798 >> v1:172.21.15.87:6790/0 conn(0x7f7654071920 legacy=0x7f7654146630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:52.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f7658d23640 1 -- 172.21.15.87:0/2467055798 shutdown_connections 2024-04-26T02:25:52.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f7658d23640 1 -- 172.21.15.87:0/2467055798 >> 172.21.15.87:0/2467055798 conn(0x7f76540b40d0 msgr2=0x7f76540b44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:52.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f7658d23640 1 -- 172.21.15.87:0/2467055798 shutdown_connections 2024-04-26T02:25:52.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.312+0000 7f7658d23640 1 -- 172.21.15.87:0/2467055798 wait complete. 2024-04-26T02:25:52.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7658d23640 1 Processor -- start 2024-04-26T02:25:52.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7658d23640 1 -- start start 2024-04-26T02:25:52.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7658d23640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7654140f70 con 0x7f7654071920 2024-04-26T02:25:52.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7658d23640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7654137a80 con 0x7f76540b0580 2024-04-26T02:25:52.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7658d23640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7654141e30 con 0x7f765414e9c0 2024-04-26T02:25:52.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7652d9d640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f765414e9c0 0x7f7654137370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38872/0 (socket says 172.21.15.87:38872) 2024-04-26T02:25:52.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7652d9d640 1 -- 172.21.15.87:0/1534295933 learned_addr learned my addr 172.21.15.87:0/1534295933 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:52.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f765259c640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f7654071920 0x7f7654140860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51790/0 (socket says 172.21.15.87:51790) 2024-04-26T02:25:52.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1557015612 0 0) 0x7f7654140f70 con 0x7f7654071920 2024-04-26T02:25:52.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7634003530 con 0x7f7654071920 2024-04-26T02:25:52.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2307738002 0 0) 0x7f7654141e30 con 0x7f765414e9c0 2024-04-26T02:25:52.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7654140f70 con 0x7f765414e9c0 2024-04-26T02:25:52.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4228009291 0 0) 0x7f7634003530 con 0x7f7654071920 2024-04-26T02:25:52.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7654141e30 con 0x7f7654071920 2024-04-26T02:25:52.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f7648001c50 con 0x7f7654071920 2024-04-26T02:25:52.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3943514892 0 0) 0x7f7654140f70 con 0x7f765414e9c0 2024-04-26T02:25:52.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7634003530 con 0x7f765414e9c0 2024-04-26T02:25:52.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f764c001c50 con 0x7f765414e9c0 2024-04-26T02:25:52.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4089367151 0 0) 0x7f7654137a80 con 0x7f76540b0580 2024-04-26T02:25:52.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7654140f70 con 0x7f76540b0580 2024-04-26T02:25:52.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3204385751 0 0) 0x7f7654141e30 con 0x7f7654071920 2024-04-26T02:25:52.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 >> v1:172.21.15.87:6790/0 conn(0x7f765414e9c0 legacy=0x7f7654137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:52.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 >> v1:172.21.15.122:6789/0 conn(0x7f76540b0580 legacy=0x7f7654136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:52.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7654142110 con 0x7f7654071920 2024-04-26T02:25:52.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7658d23640 1 -- 172.21.15.87:0/1534295933 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f76541e3140 con 0x7f7654071920 2024-04-26T02:25:52.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7658d23640 1 -- 172.21.15.87:0/1534295933 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f76541e34e0 con 0x7f7654071920 2024-04-26T02:25:52.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7648001900 con 0x7f7654071920 2024-04-26T02:25:52.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f7648004cd0 con 0x7f7654071920 2024-04-26T02:25:52.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7658d23640 1 -- 172.21.15.87:0/1534295933 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7638005090 con 0x7f7654071920 2024-04-26T02:25:52.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f764801a3f0 con 0x7f7654071920 2024-04-26T02:25:52.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.316+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f7648081220 con 0x7f7654071920 2024-04-26T02:25:52.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.320+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f7648052090 con 0x7f7654071920 2024-04-26T02:25:52.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.472+0000 7f7658d23640 1 -- 172.21.15.87:0/1534295933 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7638005380 con 0x7f7654071920 2024-04-26T02:25:52.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.472+0000 7f7642ffd640 1 -- 172.21.15.87:0/1534295933 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+14609 (unknown 3496218867 0 4259740450) 0x7f7648051990 con 0x7f7654071920 2024-04-26T02:25:52.476 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:25:52.476 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":24,"fsid":"6232cbab-48ed-4660-9de8-286f4c0e9a32","created":"2024-04-26T02:21:18.617390+0000","modified":"2024-04-26T02:25:50.571344+0000","last_up_change":"2024-04-26T02:21:28.375870+0000","last_in_change":"2024-04-26T02:21:20.860445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T02:21:27.425849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T02:21:29.621425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephsqlite","create_time":"2024-04-26T02:21:36.083447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6ea27580-6f9b-4e32-813f-fb8923bb6180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6804","nonce":815800445}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6805","nonce":815800445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6807","nonce":815800445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6806","nonce":815800445}]},"public_addr":"172.21.15.87:6804/815800445","cluster_addr":"172.21.15.87:6805/815800445","heartbeat_back_addr":"172.21.15.87:6807/815800445","heartbeat_front_addr":"172.21.15.87:6806/815800445","state":["exists","up"]},{"osd":1,"uuid":"b8afe0d9-5e03-4ae1-b91d-54b1b7071d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6800","nonce":174846219}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6801","nonce":174846219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6803","nonce":174846219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6802","nonce":174846219}]},"public_addr":"172.21.15.87:6800/174846219","cluster_addr":"172.21.15.87:6801/174846219","heartbeat_back_addr":"172.21.15.87:6803/174846219","heartbeat_front_addr":"172.21.15.87:6802/174846219","state":["exists","up"]},{"osd":2,"uuid":"d3a9234e-ef70-421d-b5ba-121f4f5cdcc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6808","nonce":2992851618}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6809","nonce":2992851618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6811","nonce":2992851618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6810","nonce":2992851618}]},"public_addr":"172.21.15.87:6808/2992851618","cluster_addr":"172.21.15.87:6809/2992851618","heartbeat_back_addr":"172.21.15.87:6811/2992851618","heartbeat_front_addr":"172.21.15.87:6810/2992851618","state":["exists","up"]},{"osd":3,"uuid":"bfc148fb-9fd7-4a94-8287-52a4f38d7100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6812","nonce":3926229128}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6813","nonce":3926229128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6815","nonce":3926229128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.87:6814","nonce":3926229128}]},"public_addr":"172.21.15.87:6812/3926229128","cluster_addr":"172.21.15.87:6813/3926229128","heartbeat_back_addr":"172.21.15.87:6815/3926229128","heartbeat_front_addr":"172.21.15.87:6814/3926229128","state":["exists","up"]},{"osd":4,"uuid":"3af86762-d77b-4c24-ac59-d0f20c4d423c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6800","nonce":850918874}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6801","nonce":850918874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6803","nonce":850918874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6802","nonce":850918874}]},"public_addr":"172.21.15.122:6800/850918874","cluster_addr":"172.21.15.122:6801/850918874","heartbeat_back_addr":"172.21.15.122:6803/850918874","heartbeat_front_addr":"172.21.15.122:6802/850918874","state":["exists","up"]},{"osd":5,"uuid":"6a28bfec-7754-4a56-bbd1-0d1d2655fde7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6812","nonce":498384109}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6813","nonce":498384109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6815","nonce":498384109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6814","nonce":498384109}]},"public_addr":"172.21.15.122:6812/498384109","cluster_addr":"172.21.15.122:6813/498384109","heartbeat_back_addr":"172.21.15.122:6815/498384109","heartbeat_front_addr":"172.21.15.122:6814/498384109","state":["exists","up"]},{"osd":6,"uuid":"db087c2e-bd80-4863-a1db-5fbd8f9fe083","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6805","nonce":662791991}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6807","nonce":662791991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6811","nonce":662791991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6810","nonce":662791991}]},"public_addr":"172.21.15.122:6805/662791991","cluster_addr":"172.21.15.122:6807/662791991","heartbeat_back_addr":"172.21.15.122:6811/662791991","heartbeat_front_addr":"172.21.15.122:6810/662791991","state":["exists","up"]},{"osd":7,"uuid":"ccf8f7a9-c44f-44f4-91d1-9f197803f67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6804","nonce":7687745}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6806","nonce":7687745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6809","nonce":7687745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.122:6808","nonce":7687745}]},"public_addr":"172.21.15.122:6804/7687745","cluster_addr":"172.21.15.122:6806/7687745","heartbeat_back_addr":"172.21.15.122:6809/7687745","heartbeat_front_addr":"172.21.15.122:6808/7687745","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.424114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.466699+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.335478+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.391653+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.462086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.410699+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:25.564818+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T02:21:26.378766+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/324966459":"2024-04-26T03:25:49.954470+0000","172.21.15.87:0/2079618140":"2024-04-26T03:24:34.494907+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:25:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.476+0000 7f7658d23640 1 -- 172.21.15.87:0/1534295933 >> v1:172.21.15.122:6816/416193625 conn(0x7f7634068410 legacy=0x7f763406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.476+0000 7f7658d23640 1 -- 172.21.15.87:0/1534295933 >> v1:172.21.15.87:6789/0 conn(0x7f7654071920 legacy=0x7f7654140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.476+0000 7f7658d23640 1 -- 172.21.15.87:0/1534295933 shutdown_connections 2024-04-26T02:25:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.476+0000 7f7658d23640 1 -- 172.21.15.87:0/1534295933 >> 172.21.15.87:0/1534295933 conn(0x7f76540b40d0 msgr2=0x7f765406f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.476+0000 7f7658d23640 1 -- 172.21.15.87:0/1534295933 shutdown_connections 2024-04-26T02:25:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:52.476+0000 7f7658d23640 1 -- 172.21.15.87:0/1534295933 wait complete. 2024-04-26T02:25:53.494 INFO:tasks.ceph:Scrubbing osd.0 2024-04-26T02:25:53.495 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-26T02:25:53.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fceea4fb640 1 Processor -- start 2024-04-26T02:25:53.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fceea4fb640 1 -- start start 2024-04-26T02:25:53.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fceea4fb640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcee406f710 con 0x7fcee413f320 2024-04-26T02:25:53.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fceea4fb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcee40a9020 con 0x7fcee406f330 2024-04-26T02:25:53.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fceea4fb640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcee4140380 con 0x7fcee4139450 2024-04-26T02:25:53.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fcee8a98640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fcee4139450 0x7fcee414ac40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38888/0 (socket says 172.21.15.87:38888) 2024-04-26T02:25:53.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fcee37fe640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fcee413f320 0x7fcee4142d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51800/0 (socket says 172.21.15.87:51800) 2024-04-26T02:25:53.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fcee8a98640 1 -- 172.21.15.87:0/2229600299 learned_addr learned my addr 172.21.15.87:0/2229600299 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:53.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fcee2ffd640 1 -- 172.21.15.87:0/2229600299 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2916706133 0 0) 0x7fcee4140380 con 0x7fcee4139450 2024-04-26T02:25:53.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fcee2ffd640 1 -- 172.21.15.87:0/2229600299 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcecc003530 con 0x7fcee4139450 2024-04-26T02:25:53.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fcee2ffd640 1 -- 172.21.15.87:0/2229600299 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2062982639 0 0) 0x7fced8001050 con 0x7fcee413f320 2024-04-26T02:25:53.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fcee2ffd640 1 -- 172.21.15.87:0/2229600299 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcee4140380 con 0x7fcee413f320 2024-04-26T02:25:53.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fcee2ffd640 1 -- 172.21.15.87:0/2229600299 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 144074270 0 0) 0x7fcecc003530 con 0x7fcee4139450 2024-04-26T02:25:53.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fcee2ffd640 1 -- 172.21.15.87:0/2229600299 >> v1:172.21.15.122:6789/0 conn(0x7fcee406f330 legacy=0x7fcee4138d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:53.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fcee2ffd640 1 -- 172.21.15.87:0/2229600299 >> v1:172.21.15.87:6789/0 conn(0x7fcee413f320 legacy=0x7fcee4142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:53.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fcee2ffd640 1 -- 172.21.15.87:0/2229600299 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee4143440 con 0x7fcee4139450 2024-04-26T02:25:53.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fcee2ffd640 1 -- 172.21.15.87:0/2229600299 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fcedc0019a0 con 0x7fcee4139450 2024-04-26T02:25:53.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fcee2ffd640 1 -- 172.21.15.87:0/2229600299 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcedc001c80 con 0x7fcee4139450 2024-04-26T02:25:53.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.628+0000 7fcee2ffd640 1 -- 172.21.15.87:0/2229600299 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fcedc002050 con 0x7fcee4139450 2024-04-26T02:25:53.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fceea4fb640 1 -- 172.21.15.87:0/2229600299 >> v1:172.21.15.87:6790/0 conn(0x7fcee4139450 legacy=0x7fcee414ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:53.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fceea4fb640 1 -- 172.21.15.87:0/2229600299 shutdown_connections 2024-04-26T02:25:53.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fceea4fb640 1 -- 172.21.15.87:0/2229600299 >> 172.21.15.87:0/2229600299 conn(0x7fcee40b4160 msgr2=0x7fcee40b4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:53.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fceea4fb640 1 -- 172.21.15.87:0/2229600299 shutdown_connections 2024-04-26T02:25:53.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fceea4fb640 1 -- 172.21.15.87:0/2229600299 wait complete. 2024-04-26T02:25:53.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fceea4fb640 1 Processor -- start 2024-04-26T02:25:53.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fceea4fb640 1 -- start start 2024-04-26T02:25:53.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fceea4fb640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcee4133ee0 con 0x7fcee413f320 2024-04-26T02:25:53.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fceea4fb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcee413d5a0 con 0x7fcee4139450 2024-04-26T02:25:53.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fceea4fb640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcee413d770 con 0x7fcee406f330 2024-04-26T02:25:53.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee3fff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fcee406f330 0x7fcee41db0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38902/0 (socket says 172.21.15.87:38902) 2024-04-26T02:25:53.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee3fff640 1 -- 172.21.15.87:0/1628663382 learned_addr learned my addr 172.21.15.87:0/1628663382 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:53.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 981040689 0 0) 0x7fcee413d770 con 0x7fcee406f330 2024-04-26T02:25:53.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fced0003530 con 0x7fcee406f330 2024-04-26T02:25:53.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3475128275 0 0) 0x7fcee4133ee0 con 0x7fcee413f320 2024-04-26T02:25:53.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcee413d770 con 0x7fcee413f320 2024-04-26T02:25:53.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3676631235 0 0) 0x7fced0003530 con 0x7fcee406f330 2024-04-26T02:25:53.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcee4133ee0 con 0x7fcee406f330 2024-04-26T02:25:53.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fced40018e0 con 0x7fcee406f330 2024-04-26T02:25:53.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1202206190 0 0) 0x7fcee413d770 con 0x7fcee413f320 2024-04-26T02:25:53.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fced0003530 con 0x7fcee413f320 2024-04-26T02:25:53.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fcedc001e30 con 0x7fcee413f320 2024-04-26T02:25:53.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 272869305 0 0) 0x7fcee413d5a0 con 0x7fcee4139450 2024-04-26T02:25:53.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcee413d770 con 0x7fcee4139450 2024-04-26T02:25:53.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3750223963 0 0) 0x7fcee4133ee0 con 0x7fcee406f330 2024-04-26T02:25:53.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 >> v1:172.21.15.122:6789/0 conn(0x7fcee4139450 legacy=0x7fcee4131f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:53.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 >> v1:172.21.15.87:6789/0 conn(0x7fcee413f320 legacy=0x7fcee41e33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:53.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee41db7b0 con 0x7fcee406f330 2024-04-26T02:25:53.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fceea4fb640 1 -- 172.21.15.87:0/1628663382 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcee41dba40 con 0x7fcee406f330 2024-04-26T02:25:53.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fceea4fb640 1 -- 172.21.15.87:0/1628663382 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcee413e1b0 con 0x7fcee406f330 2024-04-26T02:25:53.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fced4001c70 con 0x7fcee406f330 2024-04-26T02:25:53.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fced4003af0 con 0x7fcee406f330 2024-04-26T02:25:53.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.632+0000 7fceea4fb640 1 -- 172.21.15.87:0/1628663382 --> v1:172.21.15.87:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fcec4000f90 con 0x7fcee406f330 2024-04-26T02:25:53.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.636+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7fced4003c90 con 0x7fcee406f330 2024-04-26T02:25:53.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.636+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7fced4081150 con 0x7fcee406f330 2024-04-26T02:25:53.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.636+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 --> v1:172.21.15.87:6804/815800445 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fced0073e80 con 0x7fced0070210 2024-04-26T02:25:53.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.636+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (unknown 1823520817 0 0) 0x7fced40815f0 con 0x7fcee406f330 2024-04-26T02:25:53.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.636+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 <== osd.0 v1:172.21.15.87:6804/815800445 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fced0073e80 con 0x7fced0070210 2024-04-26T02:25:53.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.648+0000 7fceea4fb640 1 -- 172.21.15.87:0/1628663382 --> v1:172.21.15.87:6804/815800445 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcec4002cc0 con 0x7fced0070210 2024-04-26T02:25:53.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.648+0000 7fcee0ff9640 1 -- 172.21.15.87:0/1628663382 <== osd.0 v1:172.21.15.87:6804/815800445 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fcec4002cc0 con 0x7fced0070210 2024-04-26T02:25:53.652 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-26T02:25:53.652 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T02:25:53.652 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-26T02:25:53.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.648+0000 7fceea4fb640 1 -- 172.21.15.87:0/1628663382 >> v1:172.21.15.87:6804/815800445 conn(0x7fced0070210 legacy=0x7fced0072630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:53.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.648+0000 7fceea4fb640 1 -- 172.21.15.87:0/1628663382 >> v1:172.21.15.122:6816/416193625 conn(0x7fced0068400 legacy=0x7fced006a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:53.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.648+0000 7fceea4fb640 1 -- 172.21.15.87:0/1628663382 >> v1:172.21.15.87:6790/0 conn(0x7fcee406f330 legacy=0x7fcee41db0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:53.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.648+0000 7fcee8a98640 1 -- 172.21.15.87:0/1628663382 reap_dead start 2024-04-26T02:25:53.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.648+0000 7fceea4fb640 1 -- 172.21.15.87:0/1628663382 shutdown_connections 2024-04-26T02:25:53.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.648+0000 7fceea4fb640 1 -- 172.21.15.87:0/1628663382 >> 172.21.15.87:0/1628663382 conn(0x7fcee40b4160 msgr2=0x7fcee4071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:53.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.648+0000 7fceea4fb640 1 -- 172.21.15.87:0/1628663382 shutdown_connections 2024-04-26T02:25:53.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.648+0000 7fceea4fb640 1 -- 172.21.15.87:0/1628663382 wait complete. 2024-04-26T02:25:53.668 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-26T02:25:53.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabf3594640 1 Processor -- start 2024-04-26T02:25:53.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabf3594640 1 -- start start 2024-04-26T02:25:53.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabf3594640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabec142b20 con 0x7fabec1432c0 2024-04-26T02:25:53.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabf3594640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabec0ad060 con 0x7fabec0aad30 2024-04-26T02:25:53.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabf3594640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabec0ad340 con 0x7fabec0a8be0 2024-04-26T02:25:53.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabf1b31640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fabec0a8be0 0x7fabec14df60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38908/0 (socket says 172.21.15.87:38908) 2024-04-26T02:25:53.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabf1b31640 1 -- 172.21.15.87:0/1868135072 learned_addr learned my addr 172.21.15.87:0/1868135072 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:53.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabe3fff640 1 -- 172.21.15.87:0/1868135072 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 337412064 0 0) 0x7fabe8001120 con 0x7fabec1432c0 2024-04-26T02:25:53.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabe3fff640 1 -- 172.21.15.87:0/1868135072 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabd4003530 con 0x7fabec1432c0 2024-04-26T02:25:53.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabe3fff640 1 -- 172.21.15.87:0/1868135072 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 713812254 0 0) 0x7fabec0ad340 con 0x7fabec0a8be0 2024-04-26T02:25:53.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabe3fff640 1 -- 172.21.15.87:0/1868135072 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabe8001120 con 0x7fabec0a8be0 2024-04-26T02:25:53.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabe3fff640 1 -- 172.21.15.87:0/1868135072 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 439054782 0 0) 0x7fabd4003530 con 0x7fabec1432c0 2024-04-26T02:25:53.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabe3fff640 1 -- 172.21.15.87:0/1868135072 >> v1:172.21.15.87:6790/0 conn(0x7fabec0a8be0 legacy=0x7fabec14df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:53.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabe3fff640 1 -- 172.21.15.87:0/1868135072 >> v1:172.21.15.122:6789/0 conn(0x7fabec0aad30 legacy=0x7fabec13c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:53.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabe3fff640 1 -- 172.21.15.87:0/1868135072 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabec13c990 con 0x7fabec1432c0 2024-04-26T02:25:53.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabe3fff640 1 -- 172.21.15.87:0/1868135072 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fabe80017f0 con 0x7fabec1432c0 2024-04-26T02:25:53.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabe3fff640 1 -- 172.21.15.87:0/1868135072 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fabe8001ad0 con 0x7fabec1432c0 2024-04-26T02:25:53.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabe3fff640 1 -- 172.21.15.87:0/1868135072 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fabe8003620 con 0x7fabec1432c0 2024-04-26T02:25:53.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.768+0000 7fabf3594640 1 -- 172.21.15.87:0/1868135072 >> v1:172.21.15.87:6789/0 conn(0x7fabec1432c0 legacy=0x7fabec0a84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:53.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabf3594640 1 -- 172.21.15.87:0/1868135072 shutdown_connections 2024-04-26T02:25:53.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabf3594640 1 -- 172.21.15.87:0/1868135072 >> 172.21.15.87:0/1868135072 conn(0x7fabec0b8200 msgr2=0x7fabec0b8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:53.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabf3594640 1 -- 172.21.15.87:0/1868135072 shutdown_connections 2024-04-26T02:25:53.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabf3594640 1 -- 172.21.15.87:0/1868135072 wait complete. 2024-04-26T02:25:53.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabf3594640 1 Processor -- start 2024-04-26T02:25:53.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabf3594640 1 -- start start 2024-04-26T02:25:53.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabf3594640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabec0b0930 con 0x7fabec0aad30 2024-04-26T02:25:53.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabf3594640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabec0b1970 con 0x7fabec1432c0 2024-04-26T02:25:53.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabf3594640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabec139ea0 con 0x7fabec0a8be0 2024-04-26T02:25:53.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabf1330640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fabec0a8be0 0x7fabec0afc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38914/0 (socket says 172.21.15.87:38914) 2024-04-26T02:25:53.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabf1330640 1 -- 172.21.15.87:0/2490076894 learned_addr learned my addr 172.21.15.87:0/2490076894 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:53.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3773605121 0 0) 0x7fabec0b0930 con 0x7fabec0aad30 2024-04-26T02:25:53.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabd8003530 con 0x7fabec0aad30 2024-04-26T02:25:53.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1270733303 0 0) 0x7fabec139ea0 con 0x7fabec0a8be0 2024-04-26T02:25:53.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabec0b0930 con 0x7fabec0a8be0 2024-04-26T02:25:53.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 453404872 0 0) 0x7fabd8003530 con 0x7fabec0aad30 2024-04-26T02:25:53.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fabec139ea0 con 0x7fabec0aad30 2024-04-26T02:25:53.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 99691970 0 0) 0x7fabec0b0930 con 0x7fabec0a8be0 2024-04-26T02:25:53.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fabd8003530 con 0x7fabec0a8be0 2024-04-26T02:25:53.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fabe4001d00 con 0x7fabec0a8be0 2024-04-26T02:25:53.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fabe8003600 con 0x7fabec0aad30 2024-04-26T02:25:53.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3583838089 0 0) 0x7fabec0b1970 con 0x7fabec1432c0 2024-04-26T02:25:53.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabec0b0930 con 0x7fabec1432c0 2024-04-26T02:25:53.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1944240620 0 0) 0x7fabd8003530 con 0x7fabec0a8be0 2024-04-26T02:25:53.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 >> v1:172.21.15.122:6789/0 conn(0x7fabec1432c0 legacy=0x7fabec1da9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:53.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 >> v1:172.21.15.87:6789/0 conn(0x7fabec0aad30 legacy=0x7fabec1391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:53.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabec0b0410 con 0x7fabec0a8be0 2024-04-26T02:25:53.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabf3594640 1 -- 172.21.15.87:0/2490076894 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fabec1398f0 con 0x7fabec0a8be0 2024-04-26T02:25:53.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabf3594640 1 -- 172.21.15.87:0/2490076894 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fabec13f800 con 0x7fabec0a8be0 2024-04-26T02:25:53.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fabe4002000 con 0x7fabec0a8be0 2024-04-26T02:25:53.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fabe4003e70 con 0x7fabec0a8be0 2024-04-26T02:25:53.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabf3594640 1 -- 172.21.15.87:0/2490076894 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabcc005090 con 0x7fabec0a8be0 2024-04-26T02:25:53.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.772+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7fabe4004010 con 0x7fabec0a8be0 2024-04-26T02:25:53.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.776+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7fabe4081750 con 0x7fabec0a8be0 2024-04-26T02:25:53.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.780+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fabe40525c0 con 0x7fabec0a8be0 2024-04-26T02:25:53.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.932+0000 7fabf3594640 1 -- 172.21.15.87:0/2490076894 --> v1:172.21.15.122:6816/416193625 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fabcc000ba0 con 0x7fabd8068420 2024-04-26T02:25:53.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.932+0000 7fabe1ffb640 1 -- 172.21.15.87:0/2490076894 <== mgr.4108 v1:172.21.15.122:6816/416193625 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (unknown 2057319476 0 0) 0x7fabcc000ba0 con 0x7fabd8068420 2024-04-26T02:25:53.935 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 0 to deep-scrub 2024-04-26T02:25:53.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.936+0000 7fabf3594640 1 -- 172.21.15.87:0/2490076894 >> v1:172.21.15.122:6816/416193625 conn(0x7fabd8068420 legacy=0x7fabd806a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:53.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.936+0000 7fabf3594640 1 -- 172.21.15.87:0/2490076894 >> v1:172.21.15.87:6790/0 conn(0x7fabec0a8be0 legacy=0x7fabec0afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:53.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.936+0000 7fabf3594640 1 -- 172.21.15.87:0/2490076894 shutdown_connections 2024-04-26T02:25:53.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.936+0000 7fabf3594640 1 -- 172.21.15.87:0/2490076894 >> 172.21.15.87:0/2490076894 conn(0x7fabec0b8200 msgr2=0x7fabec074c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:53.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.936+0000 7fabf3594640 1 -- 172.21.15.87:0/2490076894 shutdown_connections 2024-04-26T02:25:53.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:53.936+0000 7fabf3594640 1 -- 172.21.15.87:0/2490076894 wait complete. 2024-04-26T02:25:53.953 INFO:tasks.ceph:Scrubbing osd.1 2024-04-26T02:25:53.953 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-26T02:25:54.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.080+0000 7f85162de640 1 Processor -- start 2024-04-26T02:25:54.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.080+0000 7f85162de640 1 -- start start 2024-04-26T02:25:54.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.080+0000 7f85162de640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85100ad2e0 con 0x7f851013c7a0 2024-04-26T02:25:54.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.080+0000 7f85162de640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8510142e60 con 0x7f85100ab0b0 2024-04-26T02:25:54.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.080+0000 7f85162de640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8510142ac0 con 0x7f85100aacd0 2024-04-26T02:25:54.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.080+0000 7f8507fff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f851013c7a0 0x7f85100a84d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51850/0 (socket says 172.21.15.87:51850) 2024-04-26T02:25:54.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.080+0000 7f850ffff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f85100aacd0 0x7f851013c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38918/0 (socket says 172.21.15.87:38918) 2024-04-26T02:25:54.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.080+0000 7f8507fff640 1 -- 172.21.15.87:0/249354267 learned_addr learned my addr 172.21.15.87:0/249354267 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:54.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850f7fe640 1 -- 172.21.15.87:0/249354267 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2494283220 0 0) 0x7f8510142ac0 con 0x7f85100aacd0 2024-04-26T02:25:54.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850f7fe640 1 -- 172.21.15.87:0/249354267 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84fc003530 con 0x7f85100aacd0 2024-04-26T02:25:54.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850f7fe640 1 -- 172.21.15.87:0/249354267 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4016931490 0 0) 0x7f85100ad2e0 con 0x7f851013c7a0 2024-04-26T02:25:54.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850f7fe640 1 -- 172.21.15.87:0/249354267 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8510142ac0 con 0x7f851013c7a0 2024-04-26T02:25:54.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850f7fe640 1 -- 172.21.15.87:0/249354267 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3504950216 0 0) 0x7f84fc003530 con 0x7f85100aacd0 2024-04-26T02:25:54.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850f7fe640 1 -- 172.21.15.87:0/249354267 >> v1:172.21.15.122:6789/0 conn(0x7f85100ab0b0 legacy=0x7f851014df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850f7fe640 1 -- 172.21.15.87:0/249354267 >> v1:172.21.15.87:6789/0 conn(0x7f851013c7a0 legacy=0x7f85100a84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850f7fe640 1 -- 172.21.15.87:0/249354267 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8510143380 con 0x7f85100aacd0 2024-04-26T02:25:54.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850f7fe640 1 -- 172.21.15.87:0/249354267 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8500001660 con 0x7f85100aacd0 2024-04-26T02:25:54.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850f7fe640 1 -- 172.21.15.87:0/249354267 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f85000018f0 con 0x7f85100aacd0 2024-04-26T02:25:54.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850f7fe640 1 -- 172.21.15.87:0/249354267 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8500001d00 con 0x7f85100aacd0 2024-04-26T02:25:54.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f85162de640 1 -- 172.21.15.87:0/249354267 >> v1:172.21.15.87:6790/0 conn(0x7f85100aacd0 legacy=0x7f851013c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f85162de640 1 -- 172.21.15.87:0/249354267 shutdown_connections 2024-04-26T02:25:54.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f85162de640 1 -- 172.21.15.87:0/249354267 >> 172.21.15.87:0/249354267 conn(0x7f85100b81a0 msgr2=0x7f85100b85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:54.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f85162de640 1 -- 172.21.15.87:0/249354267 shutdown_connections 2024-04-26T02:25:54.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f85162de640 1 -- 172.21.15.87:0/249354267 wait complete. 2024-04-26T02:25:54.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f85162de640 1 Processor -- start 2024-04-26T02:25:54.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f85162de640 1 -- start start 2024-04-26T02:25:54.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f85162de640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85100af500 con 0x7f851013c7a0 2024-04-26T02:25:54.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f85162de640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85100afe40 con 0x7f85100aacd0 2024-04-26T02:25:54.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f85162de640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85100b0010 con 0x7f85100ab0b0 2024-04-26T02:25:54.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f851487b640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f851013c7a0 0x7f85100ade10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51852/0 (socket says 172.21.15.87:51852) 2024-04-26T02:25:54.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f851487b640 1 -- 172.21.15.87:0/1457640675 learned_addr learned my addr 172.21.15.87:0/1457640675 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:54.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3856864706 0 0) 0x7f85100b0010 con 0x7f85100ab0b0 2024-04-26T02:25:54.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84f0003530 con 0x7f85100ab0b0 2024-04-26T02:25:54.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2916539750 0 0) 0x7f85100af500 con 0x7f851013c7a0 2024-04-26T02:25:54.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85100b0010 con 0x7f851013c7a0 2024-04-26T02:25:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1772128468 0 0) 0x7f84f0003530 con 0x7f85100ab0b0 2024-04-26T02:25:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85100af500 con 0x7f85100ab0b0 2024-04-26T02:25:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f84f8001780 con 0x7f85100ab0b0 2024-04-26T02:25:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.084+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1979706881 0 0) 0x7f85100b0010 con 0x7f851013c7a0 2024-04-26T02:25:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f84f0003530 con 0x7f851013c7a0 2024-04-26T02:25:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2279472265 0 0) 0x7f85100afe40 con 0x7f85100aacd0 2024-04-26T02:25:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85100b0010 con 0x7f85100aacd0 2024-04-26T02:25:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f8508001c50 con 0x7f851013c7a0 2024-04-26T02:25:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3286730486 0 0) 0x7f85100af500 con 0x7f85100ab0b0 2024-04-26T02:25:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 >> v1:172.21.15.122:6789/0 conn(0x7f85100aacd0 legacy=0x7f85101dabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 >> v1:172.21.15.87:6789/0 conn(0x7f851013c7a0 legacy=0x7f85100ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85100ae520 con 0x7f85100ab0b0 2024-04-26T02:25:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f85162de640 1 -- 172.21.15.87:0/1457640675 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f85100aea30 con 0x7f85100ab0b0 2024-04-26T02:25:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f85162de640 1 -- 172.21.15.87:0/1457640675 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8510145860 con 0x7f85100ab0b0 2024-04-26T02:25:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f84f8001d10 con 0x7f85100ab0b0 2024-04-26T02:25:54.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f84f8004a10 con 0x7f85100ab0b0 2024-04-26T02:25:54.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f85162de640 1 -- 172.21.15.87:0/1457640675 --> v1:172.21.15.87:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f84f4000f90 con 0x7f85100ab0b0 2024-04-26T02:25:54.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f84f801a130 con 0x7f85100ab0b0 2024-04-26T02:25:54.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f84f8081090 con 0x7f85100ab0b0 2024-04-26T02:25:54.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 --> v1:172.21.15.87:6800/174846219 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84f0073ea0 con 0x7f84f00702d0 2024-04-26T02:25:54.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (unknown 1823520817 0 0) 0x7f84f80814e0 con 0x7f85100ab0b0 2024-04-26T02:25:54.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.088+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 <== osd.1 v1:172.21.15.87:6800/174846219 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f84f0073ea0 con 0x7f84f00702d0 2024-04-26T02:25:54.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.104+0000 7f85162de640 1 -- 172.21.15.87:0/1457640675 --> v1:172.21.15.87:6800/174846219 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f84f4002cc0 con 0x7f84f00702d0 2024-04-26T02:25:54.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.104+0000 7f850d7fa640 1 -- 172.21.15.87:0/1457640675 <== osd.1 v1:172.21.15.87:6800/174846219 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f84f4002cc0 con 0x7f84f00702d0 2024-04-26T02:25:54.107 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-26T02:25:54.107 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T02:25:54.107 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-26T02:25:54.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.104+0000 7f85162de640 1 -- 172.21.15.87:0/1457640675 >> v1:172.21.15.87:6800/174846219 conn(0x7f84f00702d0 legacy=0x7f84f00726f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.104+0000 7f85162de640 1 -- 172.21.15.87:0/1457640675 >> v1:172.21.15.122:6816/416193625 conn(0x7f84f00684c0 legacy=0x7f84f006a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.104+0000 7f85162de640 1 -- 172.21.15.87:0/1457640675 >> v1:172.21.15.87:6790/0 conn(0x7f85100ab0b0 legacy=0x7f85100ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.104+0000 7f851487b640 1 -- 172.21.15.87:0/1457640675 reap_dead start 2024-04-26T02:25:54.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.104+0000 7f85162de640 1 -- 172.21.15.87:0/1457640675 shutdown_connections 2024-04-26T02:25:54.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.104+0000 7f85162de640 1 -- 172.21.15.87:0/1457640675 >> 172.21.15.87:0/1457640675 conn(0x7f85100b81a0 msgr2=0x7f851013fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:54.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.104+0000 7f85162de640 1 -- 172.21.15.87:0/1457640675 shutdown_connections 2024-04-26T02:25:54.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.104+0000 7f85162de640 1 -- 172.21.15.87:0/1457640675 wait complete. 2024-04-26T02:25:54.124 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-26T02:25:54.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cdcbe9640 1 Processor -- start 2024-04-26T02:25:54.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cdcbe9640 1 -- start start 2024-04-26T02:25:54.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cdcbe9640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd80ad2c0 con 0x7f3cd8143290 2024-04-26T02:25:54.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cdcbe9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd8142a40 con 0x7f3cd80a84d0 2024-04-26T02:25:54.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cdcbe9640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd80abfc0 con 0x7f3cd80aaa50 2024-04-26T02:25:54.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd659c640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f3cd80aaa50 0x7f3cd80aae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38934/0 (socket says 172.21.15.87:38934) 2024-04-26T02:25:54.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd5d9b640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f3cd8143290 0x7f3cd8138d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51860/0 (socket says 172.21.15.87:51860) 2024-04-26T02:25:54.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd659c640 1 -- 172.21.15.87:0/1895309777 learned_addr learned my addr 172.21.15.87:0/1895309777 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:54.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd559a640 1 -- 172.21.15.87:0/1895309777 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737924220 0 0) 0x7f3cd80abfc0 con 0x7f3cd80aaa50 2024-04-26T02:25:54.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd559a640 1 -- 172.21.15.87:0/1895309777 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cc0003530 con 0x7f3cd80aaa50 2024-04-26T02:25:54.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd559a640 1 -- 172.21.15.87:0/1895309777 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3838726426 0 0) 0x7f3ccc001050 con 0x7f3cd8143290 2024-04-26T02:25:54.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd559a640 1 -- 172.21.15.87:0/1895309777 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cd80abfc0 con 0x7f3cd8143290 2024-04-26T02:25:54.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd559a640 1 -- 172.21.15.87:0/1895309777 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 622866187 0 0) 0x7f3cc0003530 con 0x7f3cd80aaa50 2024-04-26T02:25:54.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd559a640 1 -- 172.21.15.87:0/1895309777 >> v1:172.21.15.122:6789/0 conn(0x7f3cd80a84d0 legacy=0x7f3cd80a8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd559a640 1 -- 172.21.15.87:0/1895309777 >> v1:172.21.15.87:6789/0 conn(0x7f3cd8143290 legacy=0x7f3cd8138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd559a640 1 -- 172.21.15.87:0/1895309777 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd80ac190 con 0x7f3cd80aaa50 2024-04-26T02:25:54.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd559a640 1 -- 172.21.15.87:0/1895309777 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3cc8001620 con 0x7f3cd80aaa50 2024-04-26T02:25:54.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd559a640 1 -- 172.21.15.87:0/1895309777 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3cc8001890 con 0x7f3cd80aaa50 2024-04-26T02:25:54.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd559a640 1 -- 172.21.15.87:0/1895309777 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3cc8001ca0 con 0x7f3cd80aaa50 2024-04-26T02:25:54.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cdcbe9640 1 -- 172.21.15.87:0/1895309777 >> v1:172.21.15.87:6790/0 conn(0x7f3cd80aaa50 legacy=0x7f3cd80aae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cdcbe9640 1 -- 172.21.15.87:0/1895309777 shutdown_connections 2024-04-26T02:25:54.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cdcbe9640 1 -- 172.21.15.87:0/1895309777 >> 172.21.15.87:0/1895309777 conn(0x7f3cd80b8180 msgr2=0x7f3cd80b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:54.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cdcbe9640 1 -- 172.21.15.87:0/1895309777 shutdown_connections 2024-04-26T02:25:54.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cdcbe9640 1 -- 172.21.15.87:0/1895309777 wait complete. 2024-04-26T02:25:54.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cdcbe9640 1 Processor -- start 2024-04-26T02:25:54.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cdcbe9640 1 -- start start 2024-04-26T02:25:54.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cdcbe9640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd81385c0 con 0x7f3cd80aaa50 2024-04-26T02:25:54.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cdcbe9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd8135210 con 0x7f3cd80a84d0 2024-04-26T02:25:54.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cdcbe9640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd8135730 con 0x7f3cd8143290 2024-04-26T02:25:54.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd6d9d640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f3cd8143290 0x7f3cd81e2d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38946/0 (socket says 172.21.15.87:38946) 2024-04-26T02:25:54.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd5d9b640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f3cd80aaa50 0x7f3cd81360d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51864/0 (socket says 172.21.15.87:51864) 2024-04-26T02:25:54.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.236+0000 7f3cd6d9d640 1 -- 172.21.15.87:0/425783599 learned_addr learned my addr 172.21.15.87:0/425783599 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:54.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 643024178 0 0) 0x7f3cd8135730 con 0x7f3cd8143290 2024-04-26T02:25:54.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cb8003530 con 0x7f3cd8143290 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1055548606 0 0) 0x7f3cd81385c0 con 0x7f3cd80aaa50 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cd8135730 con 0x7f3cd80aaa50 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 164882108 0 0) 0x7f3cb8003530 con 0x7f3cd8143290 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3cd81385c0 con 0x7f3cd8143290 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3cd0001c40 con 0x7f3cd8143290 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3919579724 0 0) 0x7f3cd8135730 con 0x7f3cd80aaa50 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3cb8003530 con 0x7f3cd80aaa50 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3ccc001660 con 0x7f3cd80aaa50 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3951773791 0 0) 0x7f3cd8135210 con 0x7f3cd80a84d0 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cd8135730 con 0x7f3cd80a84d0 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 287456393 0 0) 0x7f3cd81385c0 con 0x7f3cd8143290 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 >> v1:172.21.15.122:6789/0 conn(0x7f3cd80a84d0 legacy=0x7f3cd812fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 >> v1:172.21.15.87:6789/0 conn(0x7f3cd80aaa50 legacy=0x7f3cd81360d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd8141b00 con 0x7f3cd8143290 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cdcbe9640 1 -- 172.21.15.87:0/425783599 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd8130400 con 0x7f3cd8143290 2024-04-26T02:25:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cdcbe9640 1 -- 172.21.15.87:0/425783599 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3cd8134d80 con 0x7f3cd8143290 2024-04-26T02:25:54.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3cd00039a0 con 0x7f3cd8143290 2024-04-26T02:25:54.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3cd0003d70 con 0x7f3cd8143290 2024-04-26T02:25:54.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cdcbe9640 1 -- 172.21.15.87:0/425783599 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cbc005090 con 0x7f3cd8143290 2024-04-26T02:25:54.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f3cd0003f90 con 0x7f3cd8143290 2024-04-26T02:25:54.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.240+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f3cd0081680 con 0x7f3cd8143290 2024-04-26T02:25:54.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.244+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f3cd00524f0 con 0x7f3cd8143290 2024-04-26T02:25:54.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.392+0000 7f3cdcbe9640 1 -- 172.21.15.87:0/425783599 --> v1:172.21.15.122:6816/416193625 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f3cbc000ba0 con 0x7f3cb8068470 2024-04-26T02:25:54.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.392+0000 7f3cc6ffd640 1 -- 172.21.15.87:0/425783599 <== mgr.4108 v1:172.21.15.122:6816/416193625 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (unknown 489605429 0 0) 0x7f3cbc000ba0 con 0x7f3cb8068470 2024-04-26T02:25:54.395 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 1 to deep-scrub 2024-04-26T02:25:54.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.396+0000 7f3cdcbe9640 1 -- 172.21.15.87:0/425783599 >> v1:172.21.15.122:6816/416193625 conn(0x7f3cb8068470 legacy=0x7f3cb806a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.396+0000 7f3cdcbe9640 1 -- 172.21.15.87:0/425783599 >> v1:172.21.15.87:6790/0 conn(0x7f3cd8143290 legacy=0x7f3cd81e2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.396+0000 7f3cdcbe9640 1 -- 172.21.15.87:0/425783599 shutdown_connections 2024-04-26T02:25:54.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.396+0000 7f3cdcbe9640 1 -- 172.21.15.87:0/425783599 >> 172.21.15.87:0/425783599 conn(0x7f3cd80b8180 msgr2=0x7f3cd80b6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:54.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.396+0000 7f3cdcbe9640 1 -- 172.21.15.87:0/425783599 shutdown_connections 2024-04-26T02:25:54.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.396+0000 7f3cdcbe9640 1 -- 172.21.15.87:0/425783599 wait complete. 2024-04-26T02:25:54.412 INFO:tasks.ceph:Scrubbing osd.2 2024-04-26T02:25:54.412 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-26T02:25:54.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e53fcf640 1 Processor -- start 2024-04-26T02:25:54.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e53fcf640 1 -- start start 2024-04-26T02:25:54.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e53fcf640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e4c0af6c0 con 0x7f9e4c0af2e0 2024-04-26T02:25:54.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e53fcf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e4c142b90 con 0x7f9e4c0ac8e0 2024-04-26T02:25:54.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e53fcf640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e4c06ffd0 con 0x7f9e4c0acd20 2024-04-26T02:25:54.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e51d6b640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f9e4c0af2e0 0x7f9e4c143050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51866/0 (socket says 172.21.15.87:51866) 2024-04-26T02:25:54.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e51d6b640 1 -- 172.21.15.87:0/777546871 learned_addr learned my addr 172.21.15.87:0/777546871 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:54.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e50d69640 1 -- 172.21.15.87:0/777546871 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3698700671 0 0) 0x7f9e3c001120 con 0x7f9e4c0af2e0 2024-04-26T02:25:54.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e50d69640 1 -- 172.21.15.87:0/777546871 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e38003530 con 0x7f9e4c0af2e0 2024-04-26T02:25:54.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e50d69640 1 -- 172.21.15.87:0/777546871 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4168506730 0 0) 0x7f9e4c06ffd0 con 0x7f9e4c0acd20 2024-04-26T02:25:54.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e50d69640 1 -- 172.21.15.87:0/777546871 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e3c001120 con 0x7f9e4c0acd20 2024-04-26T02:25:54.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e50d69640 1 -- 172.21.15.87:0/777546871 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2247105842 0 0) 0x7f9e38003530 con 0x7f9e4c0af2e0 2024-04-26T02:25:54.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e50d69640 1 -- 172.21.15.87:0/777546871 >> v1:172.21.15.87:6790/0 conn(0x7f9e4c0acd20 legacy=0x7f9e4c14dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e50d69640 1 -- 172.21.15.87:0/777546871 >> v1:172.21.15.122:6789/0 conn(0x7f9e4c0ac8e0 legacy=0x7f9e4c141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e50d69640 1 -- 172.21.15.87:0/777546871 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e4c073070 con 0x7f9e4c0af2e0 2024-04-26T02:25:54.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e50d69640 1 -- 172.21.15.87:0/777546871 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9e3c0017f0 con 0x7f9e4c0af2e0 2024-04-26T02:25:54.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e50d69640 1 -- 172.21.15.87:0/777546871 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9e3c001ad0 con 0x7f9e4c0af2e0 2024-04-26T02:25:54.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e50d69640 1 -- 172.21.15.87:0/777546871 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9e3c0048b0 con 0x7f9e4c0af2e0 2024-04-26T02:25:54.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e53fcf640 1 -- 172.21.15.87:0/777546871 >> v1:172.21.15.87:6789/0 conn(0x7f9e4c0af2e0 legacy=0x7f9e4c143050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e53fcf640 1 -- 172.21.15.87:0/777546871 shutdown_connections 2024-04-26T02:25:54.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e53fcf640 1 -- 172.21.15.87:0/777546871 >> 172.21.15.87:0/777546871 conn(0x7f9e4c0b4160 msgr2=0x7f9e4c0b4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:54.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e53fcf640 1 -- 172.21.15.87:0/777546871 shutdown_connections 2024-04-26T02:25:54.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.532+0000 7f9e53fcf640 1 -- 172.21.15.87:0/777546871 wait complete. 2024-04-26T02:25:54.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e53fcf640 1 Processor -- start 2024-04-26T02:25:54.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e53fcf640 1 -- start start 2024-04-26T02:25:54.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e53fcf640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e4c11eb90 con 0x7f9e4c0ac8e0 2024-04-26T02:25:54.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e53fcf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e4c1d84b0 con 0x7f9e4c0af2e0 2024-04-26T02:25:54.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e53fcf640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e4c11d850 con 0x7f9e4c0acd20 2024-04-26T02:25:54.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e5156a640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f9e4c0acd20 0x7f9e4c1db2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38960/0 (socket says 172.21.15.87:38960) 2024-04-26T02:25:54.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e51d6b640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f9e4c0ac8e0 0x7f9e4c13f5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51882/0 (socket says 172.21.15.87:51882) 2024-04-26T02:25:54.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e5156a640 1 -- 172.21.15.87:0/1932190843 learned_addr learned my addr 172.21.15.87:0/1932190843 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:54.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 615025513 0 0) 0x7f9e4c11d850 con 0x7f9e4c0acd20 2024-04-26T02:25:54.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e30003530 con 0x7f9e4c0acd20 2024-04-26T02:25:54.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2143919650 0 0) 0x7f9e4c11eb90 con 0x7f9e4c0ac8e0 2024-04-26T02:25:54.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e4c11d850 con 0x7f9e4c0ac8e0 2024-04-26T02:25:54.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2758108496 0 0) 0x7f9e30003530 con 0x7f9e4c0acd20 2024-04-26T02:25:54.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9e4c11eb90 con 0x7f9e4c0acd20 2024-04-26T02:25:54.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9e48001940 con 0x7f9e4c0acd20 2024-04-26T02:25:54.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2377226228 0 0) 0x7f9e4c11d850 con 0x7f9e4c0ac8e0 2024-04-26T02:25:54.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9e30003530 con 0x7f9e4c0ac8e0 2024-04-26T02:25:54.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9e3c000ff0 con 0x7f9e4c0ac8e0 2024-04-26T02:25:54.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1939422365 0 0) 0x7f9e4c1d84b0 con 0x7f9e4c0af2e0 2024-04-26T02:25:54.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e4c11d850 con 0x7f9e4c0af2e0 2024-04-26T02:25:54.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3051388611 0 0) 0x7f9e4c11eb90 con 0x7f9e4c0acd20 2024-04-26T02:25:54.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 >> v1:172.21.15.122:6789/0 conn(0x7f9e4c0af2e0 legacy=0x7f9e4c13c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 >> v1:172.21.15.87:6789/0 conn(0x7f9e4c0ac8e0 legacy=0x7f9e4c13f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e4c11d220 con 0x7f9e4c0acd20 2024-04-26T02:25:54.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e53fcf640 1 -- 172.21.15.87:0/1932190843 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9e4c1dac20 con 0x7f9e4c0acd20 2024-04-26T02:25:54.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e53fcf640 1 -- 172.21.15.87:0/1932190843 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9e4c13c8c0 con 0x7f9e4c0acd20 2024-04-26T02:25:54.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9e48003660 con 0x7f9e4c0acd20 2024-04-26T02:25:54.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f9e48003a30 con 0x7f9e4c0acd20 2024-04-26T02:25:54.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e53fcf640 1 -- 172.21.15.87:0/1932190843 --> v1:172.21.15.87:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f9e34000f90 con 0x7f9e4c0acd20 2024-04-26T02:25:54.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f9e48003c50 con 0x7f9e4c0acd20 2024-04-26T02:25:54.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f9e48081310 con 0x7f9e4c0acd20 2024-04-26T02:25:54.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 --> v1:172.21.15.87:6808/2992851618 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9e30073e80 con 0x7f9e30070210 2024-04-26T02:25:54.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.536+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (unknown 1823520817 0 0) 0x7f9e48081670 con 0x7f9e4c0acd20 2024-04-26T02:25:54.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.540+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 <== osd.2 v1:172.21.15.87:6808/2992851618 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9e30073e80 con 0x7f9e30070210 2024-04-26T02:25:54.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.552+0000 7f9e53fcf640 1 -- 172.21.15.87:0/1932190843 --> v1:172.21.15.87:6808/2992851618 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9e34002cc0 con 0x7f9e30070210 2024-04-26T02:25:54.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.552+0000 7f9e427fc640 1 -- 172.21.15.87:0/1932190843 <== osd.2 v1:172.21.15.87:6808/2992851618 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f9e34002cc0 con 0x7f9e30070210 2024-04-26T02:25:54.554 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-26T02:25:54.555 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T02:25:54.555 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-26T02:25:54.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.552+0000 7f9e53fcf640 1 -- 172.21.15.87:0/1932190843 >> v1:172.21.15.87:6808/2992851618 conn(0x7f9e30070210 legacy=0x7f9e30072630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.552+0000 7f9e53fcf640 1 -- 172.21.15.87:0/1932190843 >> v1:172.21.15.122:6816/416193625 conn(0x7f9e30068400 legacy=0x7f9e3006a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.552+0000 7f9e53fcf640 1 -- 172.21.15.87:0/1932190843 >> v1:172.21.15.87:6790/0 conn(0x7f9e4c0acd20 legacy=0x7f9e4c1db2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.552+0000 7f9e5256c640 1 -- 172.21.15.87:0/1932190843 reap_dead start 2024-04-26T02:25:54.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.552+0000 7f9e53fcf640 1 -- 172.21.15.87:0/1932190843 shutdown_connections 2024-04-26T02:25:54.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.552+0000 7f9e53fcf640 1 -- 172.21.15.87:0/1932190843 >> 172.21.15.87:0/1932190843 conn(0x7f9e4c0b4160 msgr2=0x7f9e4c074c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:54.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.552+0000 7f9e53fcf640 1 -- 172.21.15.87:0/1932190843 shutdown_connections 2024-04-26T02:25:54.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.552+0000 7f9e53fcf640 1 -- 172.21.15.87:0/1932190843 wait complete. 2024-04-26T02:25:54.569 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-26T02:25:54.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c6c67d640 1 Processor -- start 2024-04-26T02:25:54.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c6c67d640 1 -- start start 2024-04-26T02:25:54.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c6c67d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c640ad2c0 con 0x7f6c64143290 2024-04-26T02:25:54.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c6c67d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c64142a40 con 0x7f6c640a84d0 2024-04-26T02:25:54.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c6c67d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c640abfc0 con 0x7f6c640aaa50 2024-04-26T02:25:54.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c69c18640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f6c64143290 0x7f6c64138d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51886/0 (socket says 172.21.15.87:51886) 2024-04-26T02:25:54.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c6a419640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f6c640aaa50 0x7f6c640aae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38970/0 (socket says 172.21.15.87:38970) 2024-04-26T02:25:54.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c69c18640 1 -- 172.21.15.87:0/2497121525 learned_addr learned my addr 172.21.15.87:0/2497121525 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:54.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c69417640 1 -- 172.21.15.87:0/2497121525 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4094080954 0 0) 0x7f6c640abfc0 con 0x7f6c640aaa50 2024-04-26T02:25:54.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c69417640 1 -- 172.21.15.87:0/2497121525 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c50003530 con 0x7f6c640aaa50 2024-04-26T02:25:54.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c69417640 1 -- 172.21.15.87:0/2497121525 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1223233696 0 0) 0x7f6c54001120 con 0x7f6c64143290 2024-04-26T02:25:54.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c69417640 1 -- 172.21.15.87:0/2497121525 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c640abfc0 con 0x7f6c64143290 2024-04-26T02:25:54.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c69417640 1 -- 172.21.15.87:0/2497121525 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2720105509 0 0) 0x7f6c50003530 con 0x7f6c640aaa50 2024-04-26T02:25:54.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c69417640 1 -- 172.21.15.87:0/2497121525 >> v1:172.21.15.122:6789/0 conn(0x7f6c640a84d0 legacy=0x7f6c640a8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c69417640 1 -- 172.21.15.87:0/2497121525 >> v1:172.21.15.87:6789/0 conn(0x7f6c64143290 legacy=0x7f6c64138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c69417640 1 -- 172.21.15.87:0/2497121525 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c640ac190 con 0x7f6c640aaa50 2024-04-26T02:25:54.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c69417640 1 -- 172.21.15.87:0/2497121525 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f6c60001620 con 0x7f6c640aaa50 2024-04-26T02:25:54.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c69417640 1 -- 172.21.15.87:0/2497121525 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6c60001880 con 0x7f6c640aaa50 2024-04-26T02:25:54.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c69417640 1 -- 172.21.15.87:0/2497121525 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f6c60001bd0 con 0x7f6c640aaa50 2024-04-26T02:25:54.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c6c67d640 1 -- 172.21.15.87:0/2497121525 >> v1:172.21.15.87:6790/0 conn(0x7f6c640aaa50 legacy=0x7f6c640aae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c6c67d640 1 -- 172.21.15.87:0/2497121525 shutdown_connections 2024-04-26T02:25:54.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c6c67d640 1 -- 172.21.15.87:0/2497121525 >> 172.21.15.87:0/2497121525 conn(0x7f6c640b8180 msgr2=0x7f6c640b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:54.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c6c67d640 1 -- 172.21.15.87:0/2497121525 shutdown_connections 2024-04-26T02:25:54.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.672+0000 7f6c6c67d640 1 -- 172.21.15.87:0/2497121525 wait complete. 2024-04-26T02:25:54.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c6c67d640 1 Processor -- start 2024-04-26T02:25:54.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c6c67d640 1 -- start start 2024-04-26T02:25:54.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c6c67d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c64138810 con 0x7f6c640aaa50 2024-04-26T02:25:54.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c6c67d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c64138af0 con 0x7f6c640a84d0 2024-04-26T02:25:54.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c6c67d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c641368a0 con 0x7f6c64143290 2024-04-26T02:25:54.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c69c18640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f6c640aaa50 0x7f6c641352d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51898/0 (socket says 172.21.15.87:51898) 2024-04-26T02:25:54.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c69c18640 1 -- 172.21.15.87:0/2413122851 learned_addr learned my addr 172.21.15.87:0/2413122851 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:54.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2201533118 0 0) 0x7f6c641368a0 con 0x7f6c64143290 2024-04-26T02:25:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c48003530 con 0x7f6c64143290 2024-04-26T02:25:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3116558689 0 0) 0x7f6c64138810 con 0x7f6c640aaa50 2024-04-26T02:25:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c641368a0 con 0x7f6c640aaa50 2024-04-26T02:25:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3434261707 0 0) 0x7f6c48003530 con 0x7f6c64143290 2024-04-26T02:25:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c64138810 con 0x7f6c64143290 2024-04-26T02:25:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3519309284 0 0) 0x7f6c641368a0 con 0x7f6c640aaa50 2024-04-26T02:25:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c48003530 con 0x7f6c640aaa50 2024-04-26T02:25:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f6c5c001b80 con 0x7f6c64143290 2024-04-26T02:25:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f6c540019f0 con 0x7f6c640aaa50 2024-04-26T02:25:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1807738562 0 0) 0x7f6c64138810 con 0x7f6c64143290 2024-04-26T02:25:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 >> v1:172.21.15.122:6789/0 conn(0x7f6c640a84d0 legacy=0x7f6c64134bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 >> v1:172.21.15.87:6789/0 conn(0x7f6c640aaa50 legacy=0x7f6c641352d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c64138f80 con 0x7f6c64143290 2024-04-26T02:25:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c6c67d640 1 -- 172.21.15.87:0/2413122851 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6c6412dcd0 con 0x7f6c64143290 2024-04-26T02:25:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c6c67d640 1 -- 172.21.15.87:0/2413122851 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6c6412f590 con 0x7f6c64143290 2024-04-26T02:25:54.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6c5c003980 con 0x7f6c64143290 2024-04-26T02:25:54.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f6c5c003d50 con 0x7f6c64143290 2024-04-26T02:25:54.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c6c67d640 1 -- 172.21.15.87:0/2413122851 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c4c005090 con 0x7f6c64143290 2024-04-26T02:25:54.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f6c5c019470 con 0x7f6c64143290 2024-04-26T02:25:54.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.676+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f6c5c080310 con 0x7f6c64143290 2024-04-26T02:25:54.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.680+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f6c5c051180 con 0x7f6c64143290 2024-04-26T02:25:54.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.828+0000 7f6c6c67d640 1 -- 172.21.15.87:0/2413122851 --> v1:172.21.15.122:6816/416193625 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f6c4c000ba0 con 0x7f6c48068400 2024-04-26T02:25:54.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.828+0000 7f6c5affd640 1 -- 172.21.15.87:0/2413122851 <== mgr.4108 v1:172.21.15.122:6816/416193625 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (unknown 3049115190 0 0) 0x7f6c4c000ba0 con 0x7f6c48068400 2024-04-26T02:25:54.832 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 2 to deep-scrub 2024-04-26T02:25:54.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.832+0000 7f6c6c67d640 1 -- 172.21.15.87:0/2413122851 >> v1:172.21.15.122:6816/416193625 conn(0x7f6c48068400 legacy=0x7f6c4806a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.832+0000 7f6c6c67d640 1 -- 172.21.15.87:0/2413122851 >> v1:172.21.15.87:6790/0 conn(0x7f6c64143290 legacy=0x7f6c641e2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.832+0000 7f6c6c67d640 1 -- 172.21.15.87:0/2413122851 shutdown_connections 2024-04-26T02:25:54.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.832+0000 7f6c6c67d640 1 -- 172.21.15.87:0/2413122851 >> 172.21.15.87:0/2413122851 conn(0x7f6c640b8180 msgr2=0x7f6c640b8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:54.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.832+0000 7f6c6c67d640 1 -- 172.21.15.87:0/2413122851 shutdown_connections 2024-04-26T02:25:54.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.832+0000 7f6c6c67d640 1 -- 172.21.15.87:0/2413122851 wait complete. 2024-04-26T02:25:54.849 INFO:tasks.ceph:Scrubbing osd.3 2024-04-26T02:25:54.849 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-26T02:25:54.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e9e4f6640 1 Processor -- start 2024-04-26T02:25:54.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e9e4f6640 1 -- start start 2024-04-26T02:25:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e9e4f6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e98073390 con 0x7f3e9813e6c0 2024-04-26T02:25:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e9e4f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e980725b0 con 0x7f3e981423c0 2024-04-26T02:25:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e9e4f6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e981427a0 con 0x7f3e980710c0 2024-04-26T02:25:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e977fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f3e980710c0 0x7f3e9813dfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38994/0 (socket says 172.21.15.87:38994) 2024-04-26T02:25:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e977fe640 1 -- 172.21.15.87:0/3484336948 learned_addr learned my addr 172.21.15.87:0/3484336948 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e96ffd640 1 -- 172.21.15.87:0/3484336948 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 183580334 0 0) 0x7f3e98073390 con 0x7f3e9813e6c0 2024-04-26T02:25:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e96ffd640 1 -- 172.21.15.87:0/3484336948 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e84003450 con 0x7f3e9813e6c0 2024-04-26T02:25:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e96ffd640 1 -- 172.21.15.87:0/3484336948 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1532570072 0 0) 0x7f3e981427a0 con 0x7f3e980710c0 2024-04-26T02:25:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e96ffd640 1 -- 172.21.15.87:0/3484336948 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e98073390 con 0x7f3e980710c0 2024-04-26T02:25:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e96ffd640 1 -- 172.21.15.87:0/3484336948 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 594605585 0 0) 0x7f3e98073390 con 0x7f3e980710c0 2024-04-26T02:25:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e96ffd640 1 -- 172.21.15.87:0/3484336948 >> v1:172.21.15.122:6789/0 conn(0x7f3e981423c0 legacy=0x7f3e980709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e96ffd640 1 -- 172.21.15.87:0/3484336948 >> v1:172.21.15.87:6789/0 conn(0x7f3e9813e6c0 legacy=0x7f3e9814a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e96ffd640 1 -- 172.21.15.87:0/3484336948 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9813fbf0 con 0x7f3e980710c0 2024-04-26T02:25:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e96ffd640 1 -- 172.21.15.87:0/3484336948 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3e80001600 con 0x7f3e980710c0 2024-04-26T02:25:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e96ffd640 1 -- 172.21.15.87:0/3484336948 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3e80001870 con 0x7f3e980710c0 2024-04-26T02:25:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e96ffd640 1 -- 172.21.15.87:0/3484336948 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3e80001c80 con 0x7f3e980710c0 2024-04-26T02:25:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/3484336948 >> v1:172.21.15.87:6790/0 conn(0x7f3e980710c0 legacy=0x7f3e9813dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/3484336948 shutdown_connections 2024-04-26T02:25:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/3484336948 >> 172.21.15.87:0/3484336948 conn(0x7f3e980b1220 msgr2=0x7f3e980b1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/3484336948 shutdown_connections 2024-04-26T02:25:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.960+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/3484336948 wait complete. 2024-04-26T02:25:54.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e9e4f6640 1 Processor -- start 2024-04-26T02:25:54.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e9e4f6640 1 -- start start 2024-04-26T02:25:54.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e9e4f6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e981354c0 con 0x7f3e9813e6c0 2024-04-26T02:25:54.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e9e4f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e980aba60 con 0x7f3e981423c0 2024-04-26T02:25:54.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e9e4f6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e98137d30 con 0x7f3e980aa7f0 2024-04-26T02:25:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e97fff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f3e9813e6c0 0x7f3e980aa0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51928/0 (socket says 172.21.15.87:51928) 2024-04-26T02:25:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e97fff640 1 -- 172.21.15.87:0/1494577342 learned_addr learned my addr 172.21.15.87:0/1494577342 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3270168854 0 0) 0x7f3e981354c0 con 0x7f3e9813e6c0 2024-04-26T02:25:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e78003530 con 0x7f3e9813e6c0 2024-04-26T02:25:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 772139008 0 0) 0x7f3e98137d30 con 0x7f3e980aa7f0 2024-04-26T02:25:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e981354c0 con 0x7f3e980aa7f0 2024-04-26T02:25:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e977fe640 1 -- 172.21.15.87:0/1494577342 >> v1:172.21.15.122:6789/0 conn(0x7f3e981423c0 legacy=0x7f3e981346a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T02:25:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e977fe640 1 -- 172.21.15.87:0/1494577342 >> v1:172.21.15.122:6789/0 conn(0x7f3e981423c0 legacy=0x7f3e981346a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T02:25:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e977fe640 1 --1- 172.21.15.87:0/1494577342 >> v1:172.21.15.122:6789/0 conn(0x7f3e981423c0 0x7f3e981346a0 :-1 s=OPENED pgs=294 cs=1 l=1).handle_message read tag failed 2024-04-26T02:25:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e977fe640 1 --1- 172.21.15.87:0/1494577342 >> v1:172.21.15.122:6789/0 conn(0x7f3e981423c0 0x7f3e981346a0 :-1 s=OPENED pgs=294 cs=1 l=1).fault on lossy channel, failing 2024-04-26T02:25:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1030504555 0 0) 0x7f3e981354c0 con 0x7f3e980aa7f0 2024-04-26T02:25:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3e98137d30 con 0x7f3e980aa7f0 2024-04-26T02:25:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2375605029 0 0) 0x7f3e78003530 con 0x7f3e9813e6c0 2024-04-26T02:25:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3e981354c0 con 0x7f3e9813e6c0 2024-04-26T02:25:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3e880012a0 con 0x7f3e980aa7f0 2024-04-26T02:25:54.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3e90001b20 con 0x7f3e9813e6c0 2024-04-26T02:25:54.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 278212960 0 0) 0x7f3e98137d30 con 0x7f3e980aa7f0 2024-04-26T02:25:54.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 >> v1:172.21.15.122:6789/0 conn(0x7f3e981423c0 legacy=0x7f3e981346a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T02:25:54.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 >> v1:172.21.15.87:6789/0 conn(0x7f3e9813e6c0 legacy=0x7f3e980aa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e98137f00 con 0x7f3e980aa7f0 2024-04-26T02:25:54.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/1494577342 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3e980b02a0 con 0x7f3e980aa7f0 2024-04-26T02:25:54.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/1494577342 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3e981d6750 con 0x7f3e980aa7f0 2024-04-26T02:25:54.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3e88003600 con 0x7f3e980aa7f0 2024-04-26T02:25:54.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3e88003950 con 0x7f3e980aa7f0 2024-04-26T02:25:54.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/1494577342 --> v1:172.21.15.87:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f3e7c000f90 con 0x7f3e980aa7f0 2024-04-26T02:25:54.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f3e88019070 con 0x7f3e980aa7f0 2024-04-26T02:25:54.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f3e8807fee0 con 0x7f3e980aa7f0 2024-04-26T02:25:54.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 --> v1:172.21.15.87:6812/3926229128 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e78073da0 con 0x7f3e780701d0 2024-04-26T02:25:54.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.964+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (unknown 1823520817 0 0) 0x7f3e880802f0 con 0x7f3e980aa7f0 2024-04-26T02:25:54.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.968+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 <== osd.3 v1:172.21.15.87:6812/3926229128 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3e78073da0 con 0x7f3e780701d0 2024-04-26T02:25:54.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.980+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/1494577342 --> v1:172.21.15.87:6812/3926229128 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3e7c002cc0 con 0x7f3e780701d0 2024-04-26T02:25:54.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.980+0000 7f3e94ff9640 1 -- 172.21.15.87:0/1494577342 <== osd.3 v1:172.21.15.87:6812/3926229128 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f3e7c002cc0 con 0x7f3e780701d0 2024-04-26T02:25:54.983 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-26T02:25:54.983 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T02:25:54.983 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-26T02:25:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.980+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/1494577342 >> v1:172.21.15.87:6812/3926229128 conn(0x7f3e780701d0 legacy=0x7f3e780725f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.980+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/1494577342 >> v1:172.21.15.122:6816/416193625 conn(0x7f3e780683c0 legacy=0x7f3e7806a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.980+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/1494577342 >> v1:172.21.15.87:6790/0 conn(0x7f3e980aa7f0 legacy=0x7f3e98134db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.980+0000 7f3e9ca93640 1 -- 172.21.15.87:0/1494577342 reap_dead start 2024-04-26T02:25:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.980+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/1494577342 shutdown_connections 2024-04-26T02:25:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.980+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/1494577342 >> 172.21.15.87:0/1494577342 conn(0x7f3e980b1220 msgr2=0x7f3e98138380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.980+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/1494577342 shutdown_connections 2024-04-26T02:25:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:54.980+0000 7f3e9e4f6640 1 -- 172.21.15.87:0/1494577342 wait complete. 2024-04-26T02:25:54.996 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-26T02:25:55.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3ea60b4640 1 Processor -- start 2024-04-26T02:25:55.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3ea60b4640 1 -- start start 2024-04-26T02:25:55.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3ea60b4640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ea00a9f00 con 0x7f3ea0135cc0 2024-04-26T02:25:55.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3ea60b4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ea00aa1e0 con 0x7f3ea01360a0 2024-04-26T02:25:55.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3ea60b4640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ea00708b0 con 0x7f3ea0072d30 2024-04-26T02:25:55.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3e9f7fe640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f3ea0135cc0 0x7f3ea00701a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51930/0 (socket says 172.21.15.87:51930) 2024-04-26T02:25:55.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3e9f7fe640 1 -- 172.21.15.87:0/3983090609 learned_addr learned my addr 172.21.15.87:0/3983090609 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:55.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3e9e7fc640 1 -- 172.21.15.87:0/3983090609 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2517472412 0 0) 0x7f3ea00a9f00 con 0x7f3ea0135cc0 2024-04-26T02:25:55.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3e9e7fc640 1 -- 172.21.15.87:0/3983090609 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e88005740 con 0x7f3ea0135cc0 2024-04-26T02:25:55.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3e9e7fc640 1 -- 172.21.15.87:0/3983090609 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4075509670 0 0) 0x7f3ea00708b0 con 0x7f3ea0072d30 2024-04-26T02:25:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3e9e7fc640 1 -- 172.21.15.87:0/3983090609 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ea00a9f00 con 0x7f3ea0072d30 2024-04-26T02:25:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3e9e7fc640 1 -- 172.21.15.87:0/3983090609 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1475888929 0 0) 0x7f3ea00a9f00 con 0x7f3ea0072d30 2024-04-26T02:25:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3e9e7fc640 1 -- 172.21.15.87:0/3983090609 >> v1:172.21.15.122:6789/0 conn(0x7f3ea01360a0 legacy=0x7f3ea0072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3e9e7fc640 1 -- 172.21.15.87:0/3983090609 >> v1:172.21.15.87:6789/0 conn(0x7f3ea0135cc0 legacy=0x7f3ea00701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3e9e7fc640 1 -- 172.21.15.87:0/3983090609 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea00a8e10 con 0x7f3ea0072d30 2024-04-26T02:25:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.100+0000 7f3e9e7fc640 1 -- 172.21.15.87:0/3983090609 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3e98001900 con 0x7f3ea0072d30 2024-04-26T02:25:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e9e7fc640 1 -- 172.21.15.87:0/3983090609 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3e98001bf0 con 0x7f3ea0072d30 2024-04-26T02:25:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e9e7fc640 1 -- 172.21.15.87:0/3983090609 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3e98002000 con 0x7f3ea0072d30 2024-04-26T02:25:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3ea60b4640 1 -- 172.21.15.87:0/3983090609 >> v1:172.21.15.87:6790/0 conn(0x7f3ea0072d30 legacy=0x7f3ea014ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3ea60b4640 1 -- 172.21.15.87:0/3983090609 shutdown_connections 2024-04-26T02:25:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3ea60b4640 1 -- 172.21.15.87:0/3983090609 >> 172.21.15.87:0/3983090609 conn(0x7f3ea006c490 msgr2=0x7f3ea006c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3ea60b4640 1 -- 172.21.15.87:0/3983090609 shutdown_connections 2024-04-26T02:25:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3ea60b4640 1 -- 172.21.15.87:0/3983090609 wait complete. 2024-04-26T02:25:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3ea60b4640 1 Processor -- start 2024-04-26T02:25:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3ea60b4640 1 -- start start 2024-04-26T02:25:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3ea60b4640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ea013c990 con 0x7f3ea01d72e0 2024-04-26T02:25:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3ea60b4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ea013cc70 con 0x7f3ea0072d30 2024-04-26T02:25:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3ea60b4640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ea01d40f0 con 0x7f3ea0135cc0 2024-04-26T02:25:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e9effd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f3ea0135cc0 0x7f3ea0132270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:39038/0 (socket says 172.21.15.87:39038) 2024-04-26T02:25:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e9effd640 1 -- 172.21.15.87:0/4033387148 learned_addr learned my addr 172.21.15.87:0/4033387148 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2779248594 0 0) 0x7f3ea01d40f0 con 0x7f3ea0135cc0 2024-04-26T02:25:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e8c003530 con 0x7f3ea0135cc0 2024-04-26T02:25:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 485977666 0 0) 0x7f3ea013c990 con 0x7f3ea01d72e0 2024-04-26T02:25:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ea01d40f0 con 0x7f3ea01d72e0 2024-04-26T02:25:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2529460096 0 0) 0x7f3ea013cc70 con 0x7f3ea0072d30 2024-04-26T02:25:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ea013c990 con 0x7f3ea0072d30 2024-04-26T02:25:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2892266654 0 0) 0x7f3ea01d40f0 con 0x7f3ea01d72e0 2024-04-26T02:25:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ea013cc70 con 0x7f3ea01d72e0 2024-04-26T02:25:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3630131249 0 0) 0x7f3e8c003530 con 0x7f3ea0135cc0 2024-04-26T02:25:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ea01d40f0 con 0x7f3ea0135cc0 2024-04-26T02:25:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3e980018c0 con 0x7f3ea01d72e0 2024-04-26T02:25:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3e94001840 con 0x7f3ea0135cc0 2024-04-26T02:25:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1475306522 0 0) 0x7f3ea013cc70 con 0x7f3ea01d72e0 2024-04-26T02:25:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 >> v1:172.21.15.87:6790/0 conn(0x7f3ea0135cc0 legacy=0x7f3ea0132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 >> v1:172.21.15.122:6789/0 conn(0x7f3ea0072d30 legacy=0x7f3ea01d39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea01d3100 con 0x7f3ea01d72e0 2024-04-26T02:25:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3ea60b4640 1 -- 172.21.15.87:0/4033387148 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3ea0132980 con 0x7f3ea01d72e0 2024-04-26T02:25:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3ea60b4640 1 -- 172.21.15.87:0/4033387148 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3ea01d6860 con 0x7f3ea01d72e0 2024-04-26T02:25:55.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3e98001bf0 con 0x7f3ea01d72e0 2024-04-26T02:25:55.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.104+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f3e98003dd0 con 0x7f3ea01d72e0 2024-04-26T02:25:55.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.108+0000 7f3ea60b4640 1 -- 172.21.15.87:0/4033387148 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e80005090 con 0x7f3ea01d72e0 2024-04-26T02:25:55.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.108+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f3e980194f0 con 0x7f3ea01d72e0 2024-04-26T02:25:55.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.108+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f3e98080250 con 0x7f3ea01d72e0 2024-04-26T02:25:55.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.112+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f3e98051100 con 0x7f3ea01d72e0 2024-04-26T02:25:55.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.272+0000 7f3ea60b4640 1 -- 172.21.15.87:0/4033387148 --> v1:172.21.15.122:6816/416193625 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f3e80000ba0 con 0x7f3e8c0683f0 2024-04-26T02:25:55.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.272+0000 7f3e87fff640 1 -- 172.21.15.87:0/4033387148 <== mgr.4108 v1:172.21.15.122:6816/416193625 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (unknown 3526570807 0 0) 0x7f3e80000ba0 con 0x7f3e8c0683f0 2024-04-26T02:25:55.277 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 3 to deep-scrub 2024-04-26T02:25:55.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.276+0000 7f3ea60b4640 1 -- 172.21.15.87:0/4033387148 >> v1:172.21.15.122:6816/416193625 conn(0x7f3e8c0683f0 legacy=0x7f3e8c06a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.276+0000 7f3ea60b4640 1 -- 172.21.15.87:0/4033387148 >> v1:172.21.15.87:6789/0 conn(0x7f3ea01d72e0 legacy=0x7f3ea01de940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.276+0000 7f3ea60b4640 1 -- 172.21.15.87:0/4033387148 shutdown_connections 2024-04-26T02:25:55.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.276+0000 7f3ea60b4640 1 -- 172.21.15.87:0/4033387148 >> 172.21.15.87:0/4033387148 conn(0x7f3ea006c490 msgr2=0x7f3ea0074c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:55.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.276+0000 7f3ea60b4640 1 -- 172.21.15.87:0/4033387148 shutdown_connections 2024-04-26T02:25:55.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.276+0000 7f3ea60b4640 1 -- 172.21.15.87:0/4033387148 wait complete. 2024-04-26T02:25:55.296 INFO:tasks.ceph:Scrubbing osd.4 2024-04-26T02:25:55.296 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-26T02:25:55.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f489acc8640 1 Processor -- start 2024-04-26T02:25:55.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f489acc8640 1 -- start start 2024-04-26T02:25:55.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f489acc8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48940a91e0 con 0x7f48940a9cb0 2024-04-26T02:25:55.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f489acc8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48940ab2a0 con 0x7f489413a680 2024-04-26T02:25:55.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f489acc8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4894140d90 con 0x7f48940a8e00 2024-04-26T02:25:55.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f4893fff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f48940a9cb0 0x7f4894071280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:60498/0 (socket says 172.21.15.87:60498) 2024-04-26T02:25:55.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f4898a64640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f48940a8e00 0x7f4894139f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40370/0 (socket says 172.21.15.87:40370) 2024-04-26T02:25:55.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f4893fff640 1 -- 172.21.15.87:0/1777279997 learned_addr learned my addr 172.21.15.87:0/1777279997 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:55.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f48937fe640 1 -- 172.21.15.87:0/1777279997 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1007768251 0 0) 0x7f4880001120 con 0x7f48940a9cb0 2024-04-26T02:25:55.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f48937fe640 1 -- 172.21.15.87:0/1777279997 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4884003530 con 0x7f48940a9cb0 2024-04-26T02:25:55.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f48937fe640 1 -- 172.21.15.87:0/1777279997 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 422095269 0 0) 0x7f4894140d90 con 0x7f48940a8e00 2024-04-26T02:25:55.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f48937fe640 1 -- 172.21.15.87:0/1777279997 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4880001120 con 0x7f48940a8e00 2024-04-26T02:25:55.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f48937fe640 1 -- 172.21.15.87:0/1777279997 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2142667714 0 0) 0x7f4884003530 con 0x7f48940a9cb0 2024-04-26T02:25:55.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f48937fe640 1 -- 172.21.15.87:0/1777279997 >> v1:172.21.15.87:6790/0 conn(0x7f48940a8e00 legacy=0x7f4894139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f48937fe640 1 -- 172.21.15.87:0/1777279997 >> v1:172.21.15.122:6789/0 conn(0x7f489413a680 legacy=0x7f489414bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f48937fe640 1 -- 172.21.15.87:0/1777279997 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48941412b0 con 0x7f48940a9cb0 2024-04-26T02:25:55.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f48937fe640 1 -- 172.21.15.87:0/1777279997 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f48800017f0 con 0x7f48940a9cb0 2024-04-26T02:25:55.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f48937fe640 1 -- 172.21.15.87:0/1777279997 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4880001a60 con 0x7f48940a9cb0 2024-04-26T02:25:55.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.416+0000 7f48937fe640 1 -- 172.21.15.87:0/1777279997 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f48800048b0 con 0x7f48940a9cb0 2024-04-26T02:25:55.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f489acc8640 1 -- 172.21.15.87:0/1777279997 >> v1:172.21.15.87:6789/0 conn(0x7f48940a9cb0 legacy=0x7f4894071280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f489acc8640 1 -- 172.21.15.87:0/1777279997 shutdown_connections 2024-04-26T02:25:55.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f489acc8640 1 -- 172.21.15.87:0/1777279997 >> 172.21.15.87:0/1777279997 conn(0x7f48940b6050 msgr2=0x7f48940b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:55.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f489acc8640 1 -- 172.21.15.87:0/1777279997 shutdown_connections 2024-04-26T02:25:55.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f489acc8640 1 -- 172.21.15.87:0/1777279997 wait complete. 2024-04-26T02:25:55.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f489acc8640 1 Processor -- start 2024-04-26T02:25:55.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f489acc8640 1 -- start start 2024-04-26T02:25:55.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f489acc8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48940ab6c0 con 0x7f48940a9cb0 2024-04-26T02:25:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f489acc8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48941d62c0 con 0x7f489413a680 2024-04-26T02:25:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f489acc8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48940abbe0 con 0x7f48940a8e00 2024-04-26T02:25:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f4893fff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f48940a9cb0 0x7f4894136360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:60510/0 (socket says 172.21.15.87:60510) 2024-04-26T02:25:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f4898a64640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f48940a8e00 0x7f48941d5240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40380/0 (socket says 172.21.15.87:40380) 2024-04-26T02:25:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f4893fff640 1 -- 172.21.15.87:0/2069153736 learned_addr learned my addr 172.21.15.87:0/2069153736 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 780337941 0 0) 0x7f48940ab6c0 con 0x7f48940a9cb0 2024-04-26T02:25:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4878003530 con 0x7f48940a9cb0 2024-04-26T02:25:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2433314118 0 0) 0x7f48940abbe0 con 0x7f48940a8e00 2024-04-26T02:25:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48940ab6c0 con 0x7f48940a8e00 2024-04-26T02:25:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3373135845 0 0) 0x7f4878003530 con 0x7f48940a9cb0 2024-04-26T02:25:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48940abbe0 con 0x7f48940a9cb0 2024-04-26T02:25:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f4880000ff0 con 0x7f48940a9cb0 2024-04-26T02:25:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2096251076 0 0) 0x7f48941d62c0 con 0x7f489413a680 2024-04-26T02:25:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4878003530 con 0x7f489413a680 2024-04-26T02:25:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1970685044 0 0) 0x7f48940ab6c0 con 0x7f48940a8e00 2024-04-26T02:25:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48941d62c0 con 0x7f48940a8e00 2024-04-26T02:25:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f4888001180 con 0x7f48940a8e00 2024-04-26T02:25:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3978831245 0 0) 0x7f48940abbe0 con 0x7f48940a9cb0 2024-04-26T02:25:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 >> v1:172.21.15.87:6790/0 conn(0x7f48940a8e00 legacy=0x7f48941d5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 >> v1:172.21.15.122:6789/0 conn(0x7f489413a680 legacy=0x7f48941e0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48940abdb0 con 0x7f48940a9cb0 2024-04-26T02:25:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f489acc8640 1 -- 172.21.15.87:0/2069153736 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f48941d5950 con 0x7f48940a9cb0 2024-04-26T02:25:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f489acc8640 1 -- 172.21.15.87:0/2069153736 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4894136c40 con 0x7f48940a9cb0 2024-04-26T02:25:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4880001a60 con 0x7f48940a9cb0 2024-04-26T02:25:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f4880004fd0 con 0x7f48940a9cb0 2024-04-26T02:25:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.420+0000 7f489acc8640 1 -- 172.21.15.87:0/2069153736 --> v1:172.21.15.87:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4874000f90 con 0x7f48940a9cb0 2024-04-26T02:25:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.424+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f488001a6f0 con 0x7f48940a9cb0 2024-04-26T02:25:55.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.424+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f4880081250 con 0x7f48940a9cb0 2024-04-26T02:25:55.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.424+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 --> v1:172.21.15.122:6800/850918874 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4878073f10 con 0x7f48780702a0 2024-04-26T02:25:55.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.424+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (unknown 1823520817 0 0) 0x7f4880081690 con 0x7f48940a9cb0 2024-04-26T02:25:55.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.424+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 <== osd.4 v1:172.21.15.122:6800/850918874 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4878073f10 con 0x7f48780702a0 2024-04-26T02:25:55.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.436+0000 7f489acc8640 1 -- 172.21.15.87:0/2069153736 --> v1:172.21.15.122:6800/850918874 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4874002cc0 con 0x7f48780702a0 2024-04-26T02:25:55.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.440+0000 7f48917fa640 1 -- 172.21.15.87:0/2069153736 <== osd.4 v1:172.21.15.122:6800/850918874 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f4874002cc0 con 0x7f48780702a0 2024-04-26T02:25:55.441 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-26T02:25:55.441 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T02:25:55.441 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-26T02:25:55.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.440+0000 7f489acc8640 1 -- 172.21.15.87:0/2069153736 >> v1:172.21.15.122:6800/850918874 conn(0x7f48780702a0 legacy=0x7f48780726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.440+0000 7f489acc8640 1 -- 172.21.15.87:0/2069153736 >> v1:172.21.15.122:6816/416193625 conn(0x7f4878068490 legacy=0x7f487806a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.440+0000 7f489acc8640 1 -- 172.21.15.87:0/2069153736 >> v1:172.21.15.87:6789/0 conn(0x7f48940a9cb0 legacy=0x7f4894136360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.440+0000 7f4899265640 1 -- 172.21.15.87:0/2069153736 reap_dead start 2024-04-26T02:25:55.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.440+0000 7f489acc8640 1 -- 172.21.15.87:0/2069153736 shutdown_connections 2024-04-26T02:25:55.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.440+0000 7f489acc8640 1 -- 172.21.15.87:0/2069153736 >> 172.21.15.87:0/2069153736 conn(0x7f48940b6050 msgr2=0x7f4894139590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:55.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.440+0000 7f489acc8640 1 -- 172.21.15.87:0/2069153736 shutdown_connections 2024-04-26T02:25:55.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.440+0000 7f489acc8640 1 -- 172.21.15.87:0/2069153736 wait complete. 2024-04-26T02:25:55.456 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-26T02:25:55.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4898f640 1 Processor -- start 2024-04-26T02:25:55.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4898f640 1 -- start start 2024-04-26T02:25:55.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4898f640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f44142500 con 0x7f7f4406cda0 2024-04-26T02:25:55.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4898f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f4406df40 con 0x7f7f4406c9c0 2024-04-26T02:25:55.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4898f640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f44142760 con 0x7f7f44143040 2024-04-26T02:25:55.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f42d9d640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f7f44143040 0x7f7f4414e080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40392/0 (socket says 172.21.15.87:40392) 2024-04-26T02:25:55.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f42d9d640 1 -- 172.21.15.87:0/3868907487 learned_addr learned my addr 172.21.15.87:0/3868907487 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:55.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f41d9b640 1 --1- 172.21.15.87:0/3868907487 >> v1:172.21.15.87:6789/0 conn(0x7f7f4406cda0 0x7f7f44141df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:60514/0 (socket says 172.21.15.87:60514) 2024-04-26T02:25:55.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f42d9d640 1 -- 172.21.15.87:0/3868907487 >> v1:172.21.15.87:6790/0 conn(0x7f7f44143040 legacy=0x7f7f4414e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-04-26T02:25:55.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f42d9d640 1 -- 172.21.15.87:0/3868907487 >> v1:172.21.15.87:6790/0 conn(0x7f7f44143040 legacy=0x7f7f4414e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T02:25:55.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f42d9d640 1 --1- 172.21.15.87:0/3868907487 >> v1:172.21.15.87:6790/0 conn(0x7f7f44143040 0x7f7f4414e080 :-1 s=OPENED pgs=330 cs=1 l=1).handle_message read tag failed 2024-04-26T02:25:55.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f42d9d640 1 --1- 172.21.15.87:0/3868907487 >> v1:172.21.15.87:6790/0 conn(0x7f7f44143040 0x7f7f4414e080 :-1 s=OPENED pgs=330 cs=1 l=1).fault on lossy channel, failing 2024-04-26T02:25:55.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4159a640 1 -- 172.21.15.87:0/3868907487 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3378400410 0 0) 0x7f7f44142500 con 0x7f7f4406cda0 2024-04-26T02:25:55.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4159a640 1 -- 172.21.15.87:0/3868907487 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f2c003530 con 0x7f7f4406cda0 2024-04-26T02:25:55.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4159a640 1 -- 172.21.15.87:0/3868907487 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2870030860 0 0) 0x7f7f2c003530 con 0x7f7f4406cda0 2024-04-26T02:25:55.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4159a640 1 -- 172.21.15.87:0/3868907487 >> v1:172.21.15.87:6790/0 conn(0x7f7f44143040 legacy=0x7f7f4414e080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T02:25:55.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4159a640 1 -- 172.21.15.87:0/3868907487 >> v1:172.21.15.122:6789/0 conn(0x7f7f4406c9c0 legacy=0x7f7f440b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4159a640 1 -- 172.21.15.87:0/3868907487 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f4406dad0 con 0x7f7f4406cda0 2024-04-26T02:25:55.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4159a640 1 -- 172.21.15.87:0/3868907487 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f7f38001620 con 0x7f7f4406cda0 2024-04-26T02:25:55.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4159a640 1 -- 172.21.15.87:0/3868907487 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7f38001880 con 0x7f7f4406cda0 2024-04-26T02:25:55.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4159a640 1 -- 172.21.15.87:0/3868907487 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f7f38001c50 con 0x7f7f4406cda0 2024-04-26T02:25:55.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4898f640 1 -- 172.21.15.87:0/3868907487 >> v1:172.21.15.87:6789/0 conn(0x7f7f4406cda0 legacy=0x7f7f44141df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4898f640 1 -- 172.21.15.87:0/3868907487 shutdown_connections 2024-04-26T02:25:55.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4898f640 1 -- 172.21.15.87:0/3868907487 >> 172.21.15.87:0/3868907487 conn(0x7f7f440b61a0 msgr2=0x7f7f440b65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:55.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4898f640 1 -- 172.21.15.87:0/3868907487 shutdown_connections 2024-04-26T02:25:55.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4898f640 1 -- 172.21.15.87:0/3868907487 wait complete. 2024-04-26T02:25:55.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4898f640 1 Processor -- start 2024-04-26T02:25:55.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.560+0000 7f7f4898f640 1 -- start start 2024-04-26T02:25:55.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f4898f640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f440b22f0 con 0x7f7f44143040 2024-04-26T02:25:55.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f4898f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f4411f280 con 0x7f7f4406cda0 2024-04-26T02:25:55.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f4898f640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f440b2750 con 0x7f7f4406c9c0 2024-04-26T02:25:55.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f4259c640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f7f4406c9c0 0x7f7f440af090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40394/0 (socket says 172.21.15.87:40394) 2024-04-26T02:25:55.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f4259c640 1 -- 172.21.15.87:0/3266707146 learned_addr learned my addr 172.21.15.87:0/3266707146 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:55.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4183634558 0 0) 0x7f7f440b22f0 con 0x7f7f44143040 2024-04-26T02:25:55.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f24003530 con 0x7f7f44143040 2024-04-26T02:25:55.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1814062431 0 0) 0x7f7f440b2750 con 0x7f7f4406c9c0 2024-04-26T02:25:55.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f440b22f0 con 0x7f7f4406c9c0 2024-04-26T02:25:55.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3642040254 0 0) 0x7f7f440b22f0 con 0x7f7f4406c9c0 2024-04-26T02:25:55.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f440b2750 con 0x7f7f4406c9c0 2024-04-26T02:25:55.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f7f34001980 con 0x7f7f4406c9c0 2024-04-26T02:25:55.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2626714171 0 0) 0x7f7f24003530 con 0x7f7f44143040 2024-04-26T02:25:55.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f440b22f0 con 0x7f7f44143040 2024-04-26T02:25:55.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f7f3c001b20 con 0x7f7f44143040 2024-04-26T02:25:55.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4234149214 0 0) 0x7f7f4411f280 con 0x7f7f4406cda0 2024-04-26T02:25:55.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f24003530 con 0x7f7f4406cda0 2024-04-26T02:25:55.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 840586945 0 0) 0x7f7f440b2750 con 0x7f7f4406c9c0 2024-04-26T02:25:55.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 >> v1:172.21.15.122:6789/0 conn(0x7f7f4406cda0 legacy=0x7f7f441d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 >> v1:172.21.15.87:6789/0 conn(0x7f7f44143040 legacy=0x7f7f440b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f4411ddc0 con 0x7f7f4406c9c0 2024-04-26T02:25:55.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f4898f640 1 -- 172.21.15.87:0/3266707146 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7f441e3010 con 0x7f7f4406c9c0 2024-04-26T02:25:55.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f4898f640 1 -- 172.21.15.87:0/3266707146 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7f441e33b0 con 0x7f7f4406c9c0 2024-04-26T02:25:55.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7f340036a0 con 0x7f7f4406c9c0 2024-04-26T02:25:55.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f7f34003a70 con 0x7f7f4406c9c0 2024-04-26T02:25:55.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f4898f640 1 -- 172.21.15.87:0/3266707146 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f28005090 con 0x7f7f4406c9c0 2024-04-26T02:25:55.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f7f34003c10 con 0x7f7f4406c9c0 2024-04-26T02:25:55.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.564+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f7f34081360 con 0x7f7f4406c9c0 2024-04-26T02:25:55.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.568+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f7f340521d0 con 0x7f7f4406c9c0 2024-04-26T02:25:55.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.732+0000 7f7f4898f640 1 -- 172.21.15.87:0/3266707146 --> v1:172.21.15.122:6816/416193625 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f7f28000ba0 con 0x7f7f240684a0 2024-04-26T02:25:55.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.732+0000 7f7f32ffd640 1 -- 172.21.15.87:0/3266707146 <== mgr.4108 v1:172.21.15.122:6816/416193625 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (unknown 3782719169 0 0) 0x7f7f28000ba0 con 0x7f7f240684a0 2024-04-26T02:25:55.734 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 4 to deep-scrub 2024-04-26T02:25:55.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.732+0000 7f7f4898f640 1 -- 172.21.15.87:0/3266707146 >> v1:172.21.15.122:6816/416193625 conn(0x7f7f240684a0 legacy=0x7f7f2406a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.732+0000 7f7f4898f640 1 -- 172.21.15.87:0/3266707146 >> v1:172.21.15.87:6790/0 conn(0x7f7f4406c9c0 legacy=0x7f7f440af090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.736+0000 7f7f4898f640 1 -- 172.21.15.87:0/3266707146 shutdown_connections 2024-04-26T02:25:55.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.736+0000 7f7f4898f640 1 -- 172.21.15.87:0/3266707146 >> 172.21.15.87:0/3266707146 conn(0x7f7f440b61a0 msgr2=0x7f7f440b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:55.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.736+0000 7f7f4898f640 1 -- 172.21.15.87:0/3266707146 shutdown_connections 2024-04-26T02:25:55.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.736+0000 7f7f4898f640 1 -- 172.21.15.87:0/3266707146 wait complete. 2024-04-26T02:25:55.753 INFO:tasks.ceph:Scrubbing osd.5 2024-04-26T02:25:55.753 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-26T02:25:55.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.872+0000 7f7693435640 1 Processor -- start 2024-04-26T02:25:55.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.872+0000 7f7693435640 1 -- start start 2024-04-26T02:25:55.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.872+0000 7f7693435640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f768c0ad4a0 con 0x7f768c06ca10 2024-04-26T02:25:55.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.872+0000 7f7693435640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f768c142b00 con 0x7f768c1432d0 2024-04-26T02:25:55.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.872+0000 7f7693435640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f768c1394e0 con 0x7f768c0b4650 2024-04-26T02:25:55.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.872+0000 7f76911d1640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f768c06ca10 0x7f768c138dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:60528/0 (socket says 172.21.15.87:60528) 2024-04-26T02:25:55.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.872+0000 7f76911d1640 1 -- 172.21.15.87:0/2463419326 learned_addr learned my addr 172.21.15.87:0/2463419326 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:55.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.872+0000 7f7683fff640 1 -- 172.21.15.87:0/2463419326 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2398392628 0 0) 0x7f768c1394e0 con 0x7f768c0b4650 2024-04-26T02:25:55.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7683fff640 1 -- 172.21.15.87:0/2463419326 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7678003530 con 0x7f768c0b4650 2024-04-26T02:25:55.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7683fff640 1 -- 172.21.15.87:0/2463419326 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4176663660 0 0) 0x7f768c0ad4a0 con 0x7f768c06ca10 2024-04-26T02:25:55.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7683fff640 1 -- 172.21.15.87:0/2463419326 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f768c1394e0 con 0x7f768c06ca10 2024-04-26T02:25:55.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7683fff640 1 -- 172.21.15.87:0/2463419326 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2568775636 0 0) 0x7f768c1394e0 con 0x7f768c06ca10 2024-04-26T02:25:55.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7683fff640 1 -- 172.21.15.87:0/2463419326 >> v1:172.21.15.87:6790/0 conn(0x7f768c0b4650 legacy=0x7f768c14df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7683fff640 1 -- 172.21.15.87:0/2463419326 >> v1:172.21.15.122:6789/0 conn(0x7f768c1432d0 legacy=0x7f768c0b3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7683fff640 1 -- 172.21.15.87:0/2463419326 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f768c06c2c0 con 0x7f768c06ca10 2024-04-26T02:25:55.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7683fff640 1 -- 172.21.15.87:0/2463419326 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f7688001620 con 0x7f768c06ca10 2024-04-26T02:25:55.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7683fff640 1 -- 172.21.15.87:0/2463419326 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7688001810 con 0x7f768c06ca10 2024-04-26T02:25:55.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7683fff640 1 -- 172.21.15.87:0/2463419326 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f7688001c20 con 0x7f768c06ca10 2024-04-26T02:25:55.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7693435640 1 -- 172.21.15.87:0/2463419326 >> v1:172.21.15.87:6789/0 conn(0x7f768c06ca10 legacy=0x7f768c138dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7693435640 1 -- 172.21.15.87:0/2463419326 shutdown_connections 2024-04-26T02:25:55.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7693435640 1 -- 172.21.15.87:0/2463419326 >> 172.21.15.87:0/2463419326 conn(0x7f768c0b6040 msgr2=0x7f768c0b6450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:55.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7693435640 1 -- 172.21.15.87:0/2463419326 shutdown_connections 2024-04-26T02:25:55.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7693435640 1 -- 172.21.15.87:0/2463419326 wait complete. 2024-04-26T02:25:55.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7693435640 1 Processor -- start 2024-04-26T02:25:55.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7693435640 1 -- start start 2024-04-26T02:25:55.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7693435640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f768c1d8730 con 0x7f768c1432d0 2024-04-26T02:25:55.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7693435640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f768c0b3650 con 0x7f768c0b4650 2024-04-26T02:25:55.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7693435640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f768c1d7820 con 0x7f768c06ca10 2024-04-26T02:25:55.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f76911d1640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f768c06ca10 0x7f768c0ae8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40406/0 (socket says 172.21.15.87:40406) 2024-04-26T02:25:55.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f76919d2640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f768c1432d0 0x7f768c0b1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:60532/0 (socket says 172.21.15.87:60532) 2024-04-26T02:25:55.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f76911d1640 1 -- 172.21.15.87:0/2380256838 learned_addr learned my addr 172.21.15.87:0/2380256838 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:55.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2812556138 0 0) 0x7f768c1d7820 con 0x7f768c06ca10 2024-04-26T02:25:55.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7670003530 con 0x7f768c06ca10 2024-04-26T02:25:55.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 460658402 0 0) 0x7f768c1d8730 con 0x7f768c1432d0 2024-04-26T02:25:55.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f768c1d7820 con 0x7f768c1432d0 2024-04-26T02:25:55.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 317766006 0 0) 0x7f7670003530 con 0x7f768c06ca10 2024-04-26T02:25:55.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f768c1d8730 con 0x7f768c06ca10 2024-04-26T02:25:55.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f7688001c20 con 0x7f768c06ca10 2024-04-26T02:25:55.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4014302233 0 0) 0x7f768c0b3650 con 0x7f768c0b4650 2024-04-26T02:25:55.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7670003530 con 0x7f768c0b4650 2024-04-26T02:25:55.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3756806332 0 0) 0x7f768c1d7820 con 0x7f768c1432d0 2024-04-26T02:25:55.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f768c0b3650 con 0x7f768c1432d0 2024-04-26T02:25:55.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f7684001760 con 0x7f768c1432d0 2024-04-26T02:25:55.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 411162443 0 0) 0x7f768c1d8730 con 0x7f768c06ca10 2024-04-26T02:25:55.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 >> v1:172.21.15.122:6789/0 conn(0x7f768c0b4650 legacy=0x7f768c1d8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 >> v1:172.21.15.87:6789/0 conn(0x7f768c1432d0 legacy=0x7f768c0b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f768c0b1b70 con 0x7f768c06ca10 2024-04-26T02:25:55.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.876+0000 7f7693435640 1 -- 172.21.15.87:0/2380256838 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f768c0b1e00 con 0x7f768c06ca10 2024-04-26T02:25:55.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.880+0000 7f7693435640 1 -- 172.21.15.87:0/2380256838 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f768c1e3590 con 0x7f768c06ca10 2024-04-26T02:25:55.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.880+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7688001810 con 0x7f768c06ca10 2024-04-26T02:25:55.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.880+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f7688004da0 con 0x7f768c06ca10 2024-04-26T02:25:55.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.880+0000 7f7693435640 1 -- 172.21.15.87:0/2380256838 --> v1:172.21.15.87:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f766c000f90 con 0x7f768c06ca10 2024-04-26T02:25:55.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.880+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f7688004f40 con 0x7f768c06ca10 2024-04-26T02:25:55.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.880+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f7688081390 con 0x7f768c06ca10 2024-04-26T02:25:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.880+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 --> v1:172.21.15.122:6812/498384109 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7670073d40 con 0x7f7670070170 2024-04-26T02:25:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.880+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (unknown 1823520817 0 0) 0x7f76880816f0 con 0x7f768c06ca10 2024-04-26T02:25:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.884+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 <== osd.5 v1:172.21.15.122:6812/498384109 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7670073d40 con 0x7f7670070170 2024-04-26T02:25:55.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.896+0000 7f7693435640 1 -- 172.21.15.87:0/2380256838 --> v1:172.21.15.122:6812/498384109 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f766c002cc0 con 0x7f7670070170 2024-04-26T02:25:55.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.896+0000 7f7681ffb640 1 -- 172.21.15.87:0/2380256838 <== osd.5 v1:172.21.15.122:6812/498384109 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f766c002cc0 con 0x7f7670070170 2024-04-26T02:25:55.900 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-26T02:25:55.900 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T02:25:55.900 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-26T02:25:55.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.896+0000 7f7693435640 1 -- 172.21.15.87:0/2380256838 >> v1:172.21.15.122:6812/498384109 conn(0x7f7670070170 legacy=0x7f7670072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.896+0000 7f7693435640 1 -- 172.21.15.87:0/2380256838 >> v1:172.21.15.122:6816/416193625 conn(0x7f7670068360 legacy=0x7f767006a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.896+0000 7f7693435640 1 -- 172.21.15.87:0/2380256838 >> v1:172.21.15.87:6790/0 conn(0x7f768c06ca10 legacy=0x7f768c0ae8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:55.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.896+0000 7f76919d2640 1 -- 172.21.15.87:0/2380256838 reap_dead start 2024-04-26T02:25:55.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.896+0000 7f7693435640 1 -- 172.21.15.87:0/2380256838 shutdown_connections 2024-04-26T02:25:55.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.896+0000 7f7693435640 1 -- 172.21.15.87:0/2380256838 >> 172.21.15.87:0/2380256838 conn(0x7f768c0b6040 msgr2=0x7f768c0728f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:55.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.896+0000 7f7693435640 1 -- 172.21.15.87:0/2380256838 shutdown_connections 2024-04-26T02:25:55.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:55.896+0000 7f7693435640 1 -- 172.21.15.87:0/2380256838 wait complete. 2024-04-26T02:25:55.918 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-26T02:25:56.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.032+0000 7fdba501b640 1 Processor -- start 2024-04-26T02:25:56.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.032+0000 7fdba501b640 1 -- start start 2024-04-26T02:25:56.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdba501b640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba013ef10 con 0x7fdba0072490 2024-04-26T02:25:56.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdba501b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba013fb50 con 0x7fdba0070970 2024-04-26T02:25:56.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdba501b640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba013f1b0 con 0x7fdba013f6d0 2024-04-26T02:25:56.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb9ed9d640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fdba0072490 0x7fdba0072870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:60546/0 (socket says 172.21.15.87:60546) 2024-04-26T02:25:56.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb9ed9d640 1 -- 172.21.15.87:0/15313842 learned_addr learned my addr 172.21.15.87:0/15313842 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:56.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb9e59c640 1 -- 172.21.15.87:0/15313842 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 892155512 0 0) 0x7fdba013ef10 con 0x7fdba0072490 2024-04-26T02:25:56.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb9e59c640 1 -- 172.21.15.87:0/15313842 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb8c004640 con 0x7fdba0072490 2024-04-26T02:25:56.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb9e59c640 1 -- 172.21.15.87:0/15313842 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3884669645 0 0) 0x7fdba013f1b0 con 0x7fdba013f6d0 2024-04-26T02:25:56.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb9e59c640 1 -- 172.21.15.87:0/15313842 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdba013ef10 con 0x7fdba013f6d0 2024-04-26T02:25:56.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb9e59c640 1 -- 172.21.15.87:0/15313842 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 896802147 0 0) 0x7fdb8c004640 con 0x7fdba0072490 2024-04-26T02:25:56.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb9e59c640 1 -- 172.21.15.87:0/15313842 >> v1:172.21.15.87:6790/0 conn(0x7fdba013f6d0 legacy=0x7fdba0135260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb9e59c640 1 -- 172.21.15.87:0/15313842 >> v1:172.21.15.122:6789/0 conn(0x7fdba0070970 legacy=0x7fdba0070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb9e59c640 1 -- 172.21.15.87:0/15313842 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba00a86e0 con 0x7fdba0072490 2024-04-26T02:25:56.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb9e59c640 1 -- 172.21.15.87:0/15313842 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fdb90001620 con 0x7fdba0072490 2024-04-26T02:25:56.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb9e59c640 1 -- 172.21.15.87:0/15313842 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdb90001940 con 0x7fdba0072490 2024-04-26T02:25:56.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb9e59c640 1 -- 172.21.15.87:0/15313842 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fdb90001d50 con 0x7fdba0072490 2024-04-26T02:25:56.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdba501b640 1 -- 172.21.15.87:0/15313842 >> v1:172.21.15.87:6789/0 conn(0x7fdba0072490 legacy=0x7fdba0072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdba501b640 1 -- 172.21.15.87:0/15313842 shutdown_connections 2024-04-26T02:25:56.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdba501b640 1 -- 172.21.15.87:0/15313842 >> 172.21.15.87:0/15313842 conn(0x7fdba00b0ff0 msgr2=0x7fdba00b1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:56.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdba501b640 1 -- 172.21.15.87:0/15313842 shutdown_connections 2024-04-26T02:25:56.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdba501b640 1 -- 172.21.15.87:0/15313842 wait complete. 2024-04-26T02:25:56.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdba501b640 1 Processor -- start 2024-04-26T02:25:56.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdba501b640 1 -- start start 2024-04-26T02:25:56.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb97fff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fdba0072490 0x7fdba01d71a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:60560/0 (socket says 172.21.15.87:60560) 2024-04-26T02:25:56.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb9ed9d640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fdba0070970 0x7fdba01d6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40422/0 (socket says 172.21.15.87:40422) 2024-04-26T02:25:56.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.036+0000 7fdb97fff640 1 -- 172.21.15.87:0/4262372581 learned_addr learned my addr 172.21.15.87:0/4262372581 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:56.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdba501b640 1 -- 172.21.15.87:0/4262372581 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba01d4410 con 0x7fdba0072490 2024-04-26T02:25:56.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdba501b640 1 -- 172.21.15.87:0/4262372581 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba0140d00 con 0x7fdba013f6d0 2024-04-26T02:25:56.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdba501b640 1 -- 172.21.15.87:0/4262372581 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba01420e0 con 0x7fdba0070970 2024-04-26T02:25:56.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3786518566 0 0) 0x7fdba01d4410 con 0x7fdba0072490 2024-04-26T02:25:56.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb80003530 con 0x7fdba0072490 2024-04-26T02:25:56.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3546064942 0 0) 0x7fdba01420e0 con 0x7fdba0070970 2024-04-26T02:25:56.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdba01d4410 con 0x7fdba0070970 2024-04-26T02:25:56.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3754145609 0 0) 0x7fdb80003530 con 0x7fdba0072490 2024-04-26T02:25:56.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdba01420e0 con 0x7fdba0072490 2024-04-26T02:25:56.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fdb88001a20 con 0x7fdba0072490 2024-04-26T02:25:56.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2092380381 0 0) 0x7fdba01d4410 con 0x7fdba0070970 2024-04-26T02:25:56.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb80003530 con 0x7fdba0070970 2024-04-26T02:25:56.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fdb90001440 con 0x7fdba0070970 2024-04-26T02:25:56.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3992046421 0 0) 0x7fdba01420e0 con 0x7fdba0072490 2024-04-26T02:25:56.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 >> v1:172.21.15.87:6790/0 conn(0x7fdba0070970 legacy=0x7fdba01d6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 >> v1:172.21.15.122:6789/0 conn(0x7fdba013f6d0 legacy=0x7fdba01ddd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba01332d0 con 0x7fdba0072490 2024-04-26T02:25:56.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdba501b640 1 -- 172.21.15.87:0/4262372581 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdba0141730 con 0x7fdba0072490 2024-04-26T02:25:56.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdba501b640 1 -- 172.21.15.87:0/4262372581 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdba01c9440 con 0x7fdba0072490 2024-04-26T02:25:56.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdb88001cd0 con 0x7fdba0072490 2024-04-26T02:25:56.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7fdb88003a50 con 0x7fdba0072490 2024-04-26T02:25:56.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdba501b640 1 -- 172.21.15.87:0/4262372581 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb84005090 con 0x7fdba0072490 2024-04-26T02:25:56.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7fdb88003c70 con 0x7fdba0072490 2024-04-26T02:25:56.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.040+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7fdb880813a0 con 0x7fdba0072490 2024-04-26T02:25:56.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.048+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fdb88052210 con 0x7fdba0072490 2024-04-26T02:25:56.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.244+0000 7fdba501b640 1 -- 172.21.15.87:0/4262372581 --> v1:172.21.15.122:6816/416193625 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fdb84000ba0 con 0x7fdb80068260 2024-04-26T02:25:56.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.244+0000 7fdb977fe640 1 -- 172.21.15.87:0/4262372581 <== mgr.4108 v1:172.21.15.122:6816/416193625 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (unknown 2264484800 0 0) 0x7fdb84000ba0 con 0x7fdb80068260 2024-04-26T02:25:56.245 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 5 to deep-scrub 2024-04-26T02:25:56.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.244+0000 7fdba501b640 1 -- 172.21.15.87:0/4262372581 >> v1:172.21.15.122:6816/416193625 conn(0x7fdb80068260 legacy=0x7fdb8006a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.244+0000 7fdba501b640 1 -- 172.21.15.87:0/4262372581 >> v1:172.21.15.87:6789/0 conn(0x7fdba0072490 legacy=0x7fdba01d71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.244+0000 7fdba501b640 1 -- 172.21.15.87:0/4262372581 shutdown_connections 2024-04-26T02:25:56.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.244+0000 7fdba501b640 1 -- 172.21.15.87:0/4262372581 >> 172.21.15.87:0/4262372581 conn(0x7fdba00b0ff0 msgr2=0x7fdba0074c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:56.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.244+0000 7fdba501b640 1 -- 172.21.15.87:0/4262372581 shutdown_connections 2024-04-26T02:25:56.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.244+0000 7fdba501b640 1 -- 172.21.15.87:0/4262372581 wait complete. 2024-04-26T02:25:56.267 INFO:tasks.ceph:Scrubbing osd.6 2024-04-26T02:25:56.267 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-26T02:25:56.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f6455541640 1 Processor -- start 2024-04-26T02:25:56.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f6455541640 1 -- start start 2024-04-26T02:25:56.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f6455541640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64500b0dc0 con 0x7f6450074ee0 2024-04-26T02:25:56.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f6455541640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6450071520 con 0x7f6450071c00 2024-04-26T02:25:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f6455541640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64500716f0 con 0x7f645013ff00 2024-04-26T02:25:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644effd640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f6450074ee0 0x7f645013f7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:60564/0 (socket says 172.21.15.87:60564) 2024-04-26T02:25:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644f7fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f645013ff00 0x7f6450144030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40426/0 (socket says 172.21.15.87:40426) 2024-04-26T02:25:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644effd640 1 -- 172.21.15.87:0/3286524914 learned_addr learned my addr 172.21.15.87:0/3286524914 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644dffb640 1 -- 172.21.15.87:0/3286524914 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3329944153 0 0) 0x7f64500b0dc0 con 0x7f6450074ee0 2024-04-26T02:25:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644dffb640 1 -- 172.21.15.87:0/3286524914 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6438003530 con 0x7f6450074ee0 2024-04-26T02:25:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644dffb640 1 -- 172.21.15.87:0/3286524914 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2533818128 0 0) 0x7f64500716f0 con 0x7f645013ff00 2024-04-26T02:25:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644dffb640 1 -- 172.21.15.87:0/3286524914 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64500b0dc0 con 0x7f645013ff00 2024-04-26T02:25:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644dffb640 1 -- 172.21.15.87:0/3286524914 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4022667334 0 0) 0x7f6450071520 con 0x7f6450071c00 2024-04-26T02:25:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644dffb640 1 -- 172.21.15.87:0/3286524914 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64500716f0 con 0x7f6450071c00 2024-04-26T02:25:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644dffb640 1 -- 172.21.15.87:0/3286524914 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2659272659 0 0) 0x7f6438003530 con 0x7f6450074ee0 2024-04-26T02:25:56.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644dffb640 1 -- 172.21.15.87:0/3286524914 >> v1:172.21.15.87:6790/0 conn(0x7f645013ff00 legacy=0x7f6450144030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644dffb640 1 -- 172.21.15.87:0/3286524914 >> v1:172.21.15.122:6789/0 conn(0x7f6450071c00 legacy=0x7f6450143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644dffb640 1 -- 172.21.15.87:0/3286524914 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f645014fd00 con 0x7f6450074ee0 2024-04-26T02:25:56.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644dffb640 1 -- 172.21.15.87:0/3286524914 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f6440001620 con 0x7f6450074ee0 2024-04-26T02:25:56.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644dffb640 1 -- 172.21.15.87:0/3286524914 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6440001940 con 0x7f6450074ee0 2024-04-26T02:25:56.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f644dffb640 1 -- 172.21.15.87:0/3286524914 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f6440001d10 con 0x7f6450074ee0 2024-04-26T02:25:56.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f6455541640 1 -- 172.21.15.87:0/3286524914 >> v1:172.21.15.87:6789/0 conn(0x7f6450074ee0 legacy=0x7f645013f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f6455541640 1 -- 172.21.15.87:0/3286524914 shutdown_connections 2024-04-26T02:25:56.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f6455541640 1 -- 172.21.15.87:0/3286524914 >> 172.21.15.87:0/3286524914 conn(0x7f645006b7b0 msgr2=0x7f64500b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:56.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f6455541640 1 -- 172.21.15.87:0/3286524914 shutdown_connections 2024-04-26T02:25:56.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.396+0000 7f6455541640 1 -- 172.21.15.87:0/3286524914 wait complete. 2024-04-26T02:25:56.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f6455541640 1 Processor -- start 2024-04-26T02:25:56.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f6455541640 1 -- start start 2024-04-26T02:25:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f6455541640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6450139360 con 0x7f645013b250 2024-04-26T02:25:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f6455541640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64501380b0 con 0x7f6450074ee0 2024-04-26T02:25:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f6455541640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f645013b800 con 0x7f645013ff00 2024-04-26T02:25:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f644e7fc640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f645013ff00 0x7f6450138540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40434/0 (socket says 172.21.15.87:40434) 2024-04-26T02:25:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f644e7fc640 1 -- 172.21.15.87:0/977725076 learned_addr learned my addr 172.21.15.87:0/977725076 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3226022016 0 0) 0x7f6450139360 con 0x7f645013b250 2024-04-26T02:25:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f643c003530 con 0x7f645013b250 2024-04-26T02:25:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 160460532 0 0) 0x7f645013b800 con 0x7f645013ff00 2024-04-26T02:25:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6450139360 con 0x7f645013ff00 2024-04-26T02:25:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 749477451 0 0) 0x7f64501380b0 con 0x7f6450074ee0 2024-04-26T02:25:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f645013b800 con 0x7f6450074ee0 2024-04-26T02:25:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1380991636 0 0) 0x7f6450139360 con 0x7f645013ff00 2024-04-26T02:25:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64501380b0 con 0x7f645013ff00 2024-04-26T02:25:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f6444001980 con 0x7f645013ff00 2024-04-26T02:25:56.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3426886223 0 0) 0x7f643c003530 con 0x7f645013b250 2024-04-26T02:25:56.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6450139360 con 0x7f645013b250 2024-04-26T02:25:56.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f64480019f0 con 0x7f645013b250 2024-04-26T02:25:56.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2375766086 0 0) 0x7f64501380b0 con 0x7f645013ff00 2024-04-26T02:25:56.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 >> v1:172.21.15.122:6789/0 conn(0x7f6450074ee0 legacy=0x7f64501356c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 >> v1:172.21.15.87:6789/0 conn(0x7f645013b250 legacy=0x7f6450138c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6450135dd0 con 0x7f645013ff00 2024-04-26T02:25:56.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f6455541640 1 -- 172.21.15.87:0/977725076 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6450136060 con 0x7f645013ff00 2024-04-26T02:25:56.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f6455541640 1 -- 172.21.15.87:0/977725076 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f645014e8b0 con 0x7f645013ff00 2024-04-26T02:25:56.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6444001ca0 con 0x7f645013ff00 2024-04-26T02:25:56.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f6444003a70 con 0x7f645013ff00 2024-04-26T02:25:56.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.400+0000 7f6455541640 1 -- 172.21.15.87:0/977725076 --> v1:172.21.15.87:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f6430000f90 con 0x7f645013ff00 2024-04-26T02:25:56.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.404+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f6444003a70 con 0x7f645013ff00 2024-04-26T02:25:56.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.404+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f6444081120 con 0x7f645013ff00 2024-04-26T02:25:56.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.404+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 --> v1:172.21.15.122:6805/662791991 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f643c073bd0 con 0x7f643c070000 2024-04-26T02:25:56.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.404+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (unknown 1823520817 0 0) 0x7f6444081530 con 0x7f645013ff00 2024-04-26T02:25:56.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.404+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 <== osd.6 v1:172.21.15.122:6805/662791991 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f643c073bd0 con 0x7f643c070000 2024-04-26T02:25:56.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.416+0000 7f6455541640 1 -- 172.21.15.87:0/977725076 --> v1:172.21.15.122:6805/662791991 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6430002cc0 con 0x7f643c070000 2024-04-26T02:25:56.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.420+0000 7f64377fe640 1 -- 172.21.15.87:0/977725076 <== osd.6 v1:172.21.15.122:6805/662791991 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f6430002cc0 con 0x7f643c070000 2024-04-26T02:25:56.421 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-26T02:25:56.421 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T02:25:56.421 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-26T02:25:56.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.420+0000 7f6455541640 1 -- 172.21.15.87:0/977725076 >> v1:172.21.15.122:6805/662791991 conn(0x7f643c070000 legacy=0x7f643c072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.420+0000 7f6455541640 1 -- 172.21.15.87:0/977725076 >> v1:172.21.15.122:6816/416193625 conn(0x7f643c0681f0 legacy=0x7f643c06a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.420+0000 7f6455541640 1 -- 172.21.15.87:0/977725076 >> v1:172.21.15.87:6790/0 conn(0x7f645013ff00 legacy=0x7f6450138540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.420+0000 7f644f7fe640 1 -- 172.21.15.87:0/977725076 reap_dead start 2024-04-26T02:25:56.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.420+0000 7f6455541640 1 -- 172.21.15.87:0/977725076 shutdown_connections 2024-04-26T02:25:56.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.420+0000 7f6455541640 1 -- 172.21.15.87:0/977725076 >> 172.21.15.87:0/977725076 conn(0x7f645006b7b0 msgr2=0x7f645013f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:56.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.420+0000 7f6455541640 1 -- 172.21.15.87:0/977725076 shutdown_connections 2024-04-26T02:25:56.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.420+0000 7f6455541640 1 -- 172.21.15.87:0/977725076 wait complete. 2024-04-26T02:25:56.439 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-26T02:25:56.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.552+0000 7f2136b2d640 1 Processor -- start 2024-04-26T02:25:56.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.552+0000 7f2136b2d640 1 -- start start 2024-04-26T02:25:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.552+0000 7f2136b2d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2130142b20 con 0x7f21300a8be0 2024-04-26T02:25:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.552+0000 7f2136b2d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21300ad060 con 0x7f21301432c0 2024-04-26T02:25:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.552+0000 7f2136b2d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21300ad340 con 0x7f21300aad30 2024-04-26T02:25:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.552+0000 7f21348c9640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f21300aad30 0x7f213013c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40446/0 (socket says 172.21.15.87:40446) 2024-04-26T02:25:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.552+0000 7f21348c9640 1 -- 172.21.15.87:0/4086372209 learned_addr learned my addr 172.21.15.87:0/4086372209 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212f7fe640 1 -- 172.21.15.87:0/4086372209 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1434415215 0 0) 0x7f21300ad340 con 0x7f21300aad30 2024-04-26T02:25:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212f7fe640 1 -- 172.21.15.87:0/4086372209 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2120003530 con 0x7f21300aad30 2024-04-26T02:25:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212f7fe640 1 -- 172.21.15.87:0/4086372209 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4031217307 0 0) 0x7f21280014a0 con 0x7f21300a8be0 2024-04-26T02:25:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212f7fe640 1 -- 172.21.15.87:0/4086372209 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21300ad340 con 0x7f21300a8be0 2024-04-26T02:25:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212f7fe640 1 -- 172.21.15.87:0/4086372209 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3248152226 0 0) 0x7f2120003530 con 0x7f21300aad30 2024-04-26T02:25:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212f7fe640 1 -- 172.21.15.87:0/4086372209 >> v1:172.21.15.122:6789/0 conn(0x7f21301432c0 legacy=0x7f21300a84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212f7fe640 1 -- 172.21.15.87:0/4086372209 >> v1:172.21.15.87:6789/0 conn(0x7f21300a8be0 legacy=0x7f213014df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212f7fe640 1 -- 172.21.15.87:0/4086372209 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f213013c990 con 0x7f21300aad30 2024-04-26T02:25:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212f7fe640 1 -- 172.21.15.87:0/4086372209 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f2124001620 con 0x7f21300aad30 2024-04-26T02:25:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212f7fe640 1 -- 172.21.15.87:0/4086372209 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2124001880 con 0x7f21300aad30 2024-04-26T02:25:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212f7fe640 1 -- 172.21.15.87:0/4086372209 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f2124001c50 con 0x7f21300aad30 2024-04-26T02:25:56.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f2136b2d640 1 -- 172.21.15.87:0/4086372209 >> v1:172.21.15.87:6790/0 conn(0x7f21300aad30 legacy=0x7f213013c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f2136b2d640 1 -- 172.21.15.87:0/4086372209 shutdown_connections 2024-04-26T02:25:56.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f2136b2d640 1 -- 172.21.15.87:0/4086372209 >> 172.21.15.87:0/4086372209 conn(0x7f21300b8200 msgr2=0x7f21300b8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:56.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f2136b2d640 1 -- 172.21.15.87:0/4086372209 shutdown_connections 2024-04-26T02:25:56.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f2136b2d640 1 -- 172.21.15.87:0/4086372209 wait complete. 2024-04-26T02:25:56.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f2136b2d640 1 Processor -- start 2024-04-26T02:25:56.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f2136b2d640 1 -- start start 2024-04-26T02:25:56.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f2136b2d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2130136bf0 con 0x7f21300aad30 2024-04-26T02:25:56.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f2136b2d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21301467c0 con 0x7f21300a8be0 2024-04-26T02:25:56.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f2136b2d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21300ad890 con 0x7f21301432c0 2024-04-26T02:25:56.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f21350ca640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f21301432c0 0x7f21301364e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40448/0 (socket says 172.21.15.87:40448) 2024-04-26T02:25:56.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212ffff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f21300aad30 0x7f2130135dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:60602/0 (socket says 172.21.15.87:60602) 2024-04-26T02:25:56.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f21350ca640 1 -- 172.21.15.87:0/3366939478 learned_addr learned my addr 172.21.15.87:0/3366939478 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 945590179 0 0) 0x7f21300ad890 con 0x7f21301432c0 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2114003530 con 0x7f21301432c0 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 558798849 0 0) 0x7f2130136bf0 con 0x7f21300aad30 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21300ad890 con 0x7f21300aad30 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3391080073 0 0) 0x7f2114003530 con 0x7f21301432c0 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2130136bf0 con 0x7f21301432c0 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3128409831 0 0) 0x7f21300ad890 con 0x7f21300aad30 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2114003530 con 0x7f21300aad30 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f2128001930 con 0x7f21301432c0 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1218727568 0 0) 0x7f21301467c0 con 0x7f21300a8be0 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21300ad890 con 0x7f21300a8be0 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.556+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f211c001820 con 0x7f21300aad30 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2463121586 0 0) 0x7f2130136bf0 con 0x7f21301432c0 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 >> v1:172.21.15.122:6789/0 conn(0x7f21300a8be0 legacy=0x7f21301dab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 >> v1:172.21.15.87:6789/0 conn(0x7f21300aad30 legacy=0x7f2130135dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2130145bd0 con 0x7f21301432c0 2024-04-26T02:25:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f2136b2d640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f21301452d0 con 0x7f21301432c0 2024-04-26T02:25:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f2136b2d640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f21301db240 con 0x7f21301432c0 2024-04-26T02:25:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2128001b60 con 0x7f21301432c0 2024-04-26T02:25:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f2128003dd0 con 0x7f21301432c0 2024-04-26T02:25:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f2136b2d640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f213004fd20 con 0x7f21301432c0 2024-04-26T02:25:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f2128003f70 con 0x7f21301432c0 2024-04-26T02:25:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f21350ca640 1 -- 172.21.15.87:0/3366939478 >> v1:172.21.15.87:6790/0 conn(0x7f21301432c0 legacy=0x7f21301364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T02:25:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f21350ca640 1 -- 172.21.15.87:0/3366939478 >> v1:172.21.15.87:6790/0 conn(0x7f21301432c0 legacy=0x7f21301364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T02:25:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f21350ca640 1 --1- 172.21.15.87:0/3366939478 >> v1:172.21.15.87:6790/0 conn(0x7f21301432c0 0x7f21301364e0 :-1 s=OPENED pgs=339 cs=1 l=1).handle_message read tag failed 2024-04-26T02:25:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f21350ca640 1 --1- 172.21.15.87:0/3366939478 >> v1:172.21.15.87:6790/0 conn(0x7f21301432c0 0x7f21301364e0 :-1 s=OPENED pgs=339 cs=1 l=1).fault on lossy channel, failing 2024-04-26T02:25:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 >> v1:172.21.15.87:6790/0 conn(0x7f21301432c0 legacy=0x7f21301364e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T02:25:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2130136bf0 con 0x7f21300aad30 2024-04-26T02:25:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f21301467c0 con 0x7f21300a8be0 2024-04-26T02:25:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f21140795c0 con 0x7f211406c170 2024-04-26T02:25:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2516684236 0 0) 0x7f2130136bf0 con 0x7f21300aad30 2024-04-26T02:25:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.87:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f2114077560 con 0x7f21300aad30 2024-04-26T02:25:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 3145390903 0 0) 0x7f2114077560 con 0x7f21300aad30 2024-04-26T02:25:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 >> v1:172.21.15.87:6790/0 conn(0x7f211406c170 legacy=0x7f211406e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 >> v1:172.21.15.122:6789/0 conn(0x7f21300a8be0 legacy=0x7f2114071ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f211407a7c0 con 0x7f21300aad30 2024-04-26T02:25:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2130136bf0 con 0x7f21300aad30 2024-04-26T02:25:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f2124001230 con 0x7f21300aad30 2024-04-26T02:25:56.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.560+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2124001110 con 0x7f21300aad30 2024-04-26T02:25:56.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.564+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f2124001450 con 0x7f21300aad30 2024-04-26T02:25:56.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.564+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.0 v1:172.21.15.87:6789/0 6 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f2124082870 con 0x7f21300aad30 2024-04-26T02:25:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.568+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f2124053820 con 0x7f21300aad30 2024-04-26T02:25:56.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.728+0000 7f2136b2d640 1 -- 172.21.15.87:0/3366939478 --> v1:172.21.15.122:6816/416193625 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f2130145da0 con 0x7f21140684a0 2024-04-26T02:25:56.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.732+0000 7f212d7fa640 1 -- 172.21.15.87:0/3366939478 <== mgr.4108 v1:172.21.15.122:6816/416193625 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (unknown 778718403 0 0) 0x7f2130145da0 con 0x7f21140684a0 2024-04-26T02:25:56.733 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 6 to deep-scrub 2024-04-26T02:25:56.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.732+0000 7f2136b2d640 1 -- 172.21.15.87:0/3366939478 >> v1:172.21.15.122:6816/416193625 conn(0x7f21140684a0 legacy=0x7f211406a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.732+0000 7f2136b2d640 1 -- 172.21.15.87:0/3366939478 >> v1:172.21.15.87:6789/0 conn(0x7f21300aad30 legacy=0x7f2114075a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.732+0000 7f2136b2d640 1 -- 172.21.15.87:0/3366939478 shutdown_connections 2024-04-26T02:25:56.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.732+0000 7f2136b2d640 1 -- 172.21.15.87:0/3366939478 >> 172.21.15.87:0/3366939478 conn(0x7f21300b8200 msgr2=0x7f213013a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:56.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.732+0000 7f2136b2d640 1 -- 172.21.15.87:0/3366939478 shutdown_connections 2024-04-26T02:25:56.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.732+0000 7f2136b2d640 1 -- 172.21.15.87:0/3366939478 wait complete. 2024-04-26T02:25:56.752 INFO:tasks.ceph:Scrubbing osd.7 2024-04-26T02:25:56.752 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-26T02:25:56.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.868+0000 7f1c7f292640 1 Processor -- start 2024-04-26T02:25:56.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.868+0000 7f1c7f292640 1 -- start start 2024-04-26T02:25:56.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.868+0000 7f1c7f292640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c780a9360 con 0x7f1c780a9e30 2024-04-26T02:25:56.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.868+0000 7f1c7f292640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c780ab420 con 0x7f1c7813a7d0 2024-04-26T02:25:56.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.868+0000 7f1c7f292640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c78140cc0 con 0x7f1c780a8f80 2024-04-26T02:25:56.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c7d02e640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f1c780a8f80 0x7f1c780b3ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40464/0 (socket says 172.21.15.87:40464) 2024-04-26T02:25:56.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c7c82d640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f1c780a9e30 0x7f1c7813a0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:60618/0 (socket says 172.21.15.87:60618) 2024-04-26T02:25:56.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c7d02e640 1 -- 172.21.15.87:0/3464182592 learned_addr learned my addr 172.21.15.87:0/3464182592 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:56.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c6ffff640 1 -- 172.21.15.87:0/3464182592 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2538691019 0 0) 0x7f1c78140cc0 con 0x7f1c780a8f80 2024-04-26T02:25:56.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c6ffff640 1 -- 172.21.15.87:0/3464182592 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c64003530 con 0x7f1c780a8f80 2024-04-26T02:25:56.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c6ffff640 1 -- 172.21.15.87:0/3464182592 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1976996114 0 0) 0x7f1c68001120 con 0x7f1c780a9e30 2024-04-26T02:25:56.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c6ffff640 1 -- 172.21.15.87:0/3464182592 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c78140cc0 con 0x7f1c780a9e30 2024-04-26T02:25:56.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c6ffff640 1 -- 172.21.15.87:0/3464182592 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 901955978 0 0) 0x7f1c64003530 con 0x7f1c780a8f80 2024-04-26T02:25:56.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c6ffff640 1 -- 172.21.15.87:0/3464182592 >> v1:172.21.15.122:6789/0 conn(0x7f1c7813a7d0 legacy=0x7f1c7814bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c6ffff640 1 -- 172.21.15.87:0/3464182592 >> v1:172.21.15.87:6789/0 conn(0x7f1c780a9e30 legacy=0x7f1c7813a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c6ffff640 1 -- 172.21.15.87:0/3464182592 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c780aa2a0 con 0x7f1c780a8f80 2024-04-26T02:25:56.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c6ffff640 1 -- 172.21.15.87:0/3464182592 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f1c74001620 con 0x7f1c780a8f80 2024-04-26T02:25:56.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c6ffff640 1 -- 172.21.15.87:0/3464182592 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1c740018c0 con 0x7f1c780a8f80 2024-04-26T02:25:56.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c6ffff640 1 -- 172.21.15.87:0/3464182592 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f1c74001cd0 con 0x7f1c780a8f80 2024-04-26T02:25:56.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c7f292640 1 -- 172.21.15.87:0/3464182592 >> v1:172.21.15.87:6790/0 conn(0x7f1c780a8f80 legacy=0x7f1c780b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c7f292640 1 -- 172.21.15.87:0/3464182592 shutdown_connections 2024-04-26T02:25:56.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c7f292640 1 -- 172.21.15.87:0/3464182592 >> 172.21.15.87:0/3464182592 conn(0x7f1c780b61a0 msgr2=0x7f1c780b65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:56.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c7f292640 1 -- 172.21.15.87:0/3464182592 shutdown_connections 2024-04-26T02:25:56.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c7f292640 1 -- 172.21.15.87:0/3464182592 wait complete. 2024-04-26T02:25:56.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c7f292640 1 Processor -- start 2024-04-26T02:25:56.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.872+0000 7f1c7f292640 1 -- start start 2024-04-26T02:25:56.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c7f292640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c7813d1d0 con 0x7f1c780a9e30 2024-04-26T02:25:56.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c7f292640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c781d5b90 con 0x7f1c7813a7d0 2024-04-26T02:25:56.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c7f292640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c781d5d60 con 0x7f1c780a8f80 2024-04-26T02:25:56.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c7d02e640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f1c780a8f80 0x7f1c781d5480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40472/0 (socket says 172.21.15.87:40472) 2024-04-26T02:25:56.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c7c82d640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f1c780a9e30 0x7f1c781382a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:60624/0 (socket says 172.21.15.87:60624) 2024-04-26T02:25:56.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c7d02e640 1 -- 172.21.15.87:0/2703524698 learned_addr learned my addr 172.21.15.87:0/2703524698 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:56.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3389532380 0 0) 0x7f1c780aa2a0 con 0x7f1c780a8f80 2024-04-26T02:25:56.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c5c003530 con 0x7f1c780a8f80 2024-04-26T02:25:56.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3252018218 0 0) 0x7f1c7813d1d0 con 0x7f1c780a9e30 2024-04-26T02:25:56.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c780aa2a0 con 0x7f1c780a9e30 2024-04-26T02:25:56.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 172718760 0 0) 0x7f1c781d5b90 con 0x7f1c7813a7d0 2024-04-26T02:25:56.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c7813d1d0 con 0x7f1c7813a7d0 2024-04-26T02:25:56.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2033888485 0 0) 0x7f1c5c003530 con 0x7f1c780a8f80 2024-04-26T02:25:56.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c781d5b90 con 0x7f1c780a8f80 2024-04-26T02:25:56.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f1c74001520 con 0x7f1c780a8f80 2024-04-26T02:25:56.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1006102903 0 0) 0x7f1c780aa2a0 con 0x7f1c780a9e30 2024-04-26T02:25:56.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c5c003530 con 0x7f1c780a9e30 2024-04-26T02:25:56.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f1c68001550 con 0x7f1c780a9e30 2024-04-26T02:25:56.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1427282721 0 0) 0x7f1c781d5b90 con 0x7f1c780a8f80 2024-04-26T02:25:56.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 >> v1:172.21.15.122:6789/0 conn(0x7f1c7813a7d0 legacy=0x7f1c781e0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 >> v1:172.21.15.87:6789/0 conn(0x7f1c780a9e30 legacy=0x7f1c781382a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c78139a20 con 0x7f1c780a8f80 2024-04-26T02:25:56.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c7f292640 1 -- 172.21.15.87:0/2703524698 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1c7813d9d0 con 0x7f1c780a8f80 2024-04-26T02:25:56.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c7f292640 1 -- 172.21.15.87:0/2703524698 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1c78138f80 con 0x7f1c780a8f80 2024-04-26T02:25:56.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1c74004c00 con 0x7f1c780a8f80 2024-04-26T02:25:56.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f1c74004db0 con 0x7f1c780a8f80 2024-04-26T02:25:56.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c7f292640 1 -- 172.21.15.87:0/2703524698 --> v1:172.21.15.87:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f1c58000f90 con 0x7f1c780a8f80 2024-04-26T02:25:56.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.876+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f1c7401a4d0 con 0x7f1c780a8f80 2024-04-26T02:25:56.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.880+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f1c740811f0 con 0x7f1c780a8f80 2024-04-26T02:25:56.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.880+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 --> v1:172.21.15.122:6804/7687745 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c740018c0 con 0x7f1c5c0703c0 2024-04-26T02:25:56.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.880+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (unknown 1823520817 0 0) 0x7f1c74081600 con 0x7f1c780a8f80 2024-04-26T02:25:56.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.880+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 <== osd.7 v1:172.21.15.122:6804/7687745 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1c740018c0 con 0x7f1c5c0703c0 2024-04-26T02:25:56.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.896+0000 7f1c7f292640 1 -- 172.21.15.87:0/2703524698 --> v1:172.21.15.122:6804/7687745 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1c58002cc0 con 0x7f1c5c0703c0 2024-04-26T02:25:56.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.900+0000 7f1c6dffb640 1 -- 172.21.15.87:0/2703524698 <== osd.7 v1:172.21.15.122:6804/7687745 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f1c58002cc0 con 0x7f1c5c0703c0 2024-04-26T02:25:56.901 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-26T02:25:56.901 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T02:25:56.901 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-26T02:25:56.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.900+0000 7f1c7f292640 1 -- 172.21.15.87:0/2703524698 >> v1:172.21.15.122:6804/7687745 conn(0x7f1c5c0703c0 legacy=0x7f1c5c0727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.900+0000 7f1c7f292640 1 -- 172.21.15.87:0/2703524698 >> v1:172.21.15.122:6816/416193625 conn(0x7f1c5c0685b0 legacy=0x7f1c5c06aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.900+0000 7f1c7f292640 1 -- 172.21.15.87:0/2703524698 >> v1:172.21.15.87:6790/0 conn(0x7f1c780a8f80 legacy=0x7f1c781d5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:56.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.900+0000 7f1c7d82f640 1 -- 172.21.15.87:0/2703524698 reap_dead start 2024-04-26T02:25:56.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.900+0000 7f1c7f292640 1 -- 172.21.15.87:0/2703524698 shutdown_connections 2024-04-26T02:25:56.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.900+0000 7f1c7f292640 1 -- 172.21.15.87:0/2703524698 >> 172.21.15.87:0/2703524698 conn(0x7f1c780b61a0 msgr2=0x7f1c781c7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:56.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.900+0000 7f1c7f292640 1 -- 172.21.15.87:0/2703524698 shutdown_connections 2024-04-26T02:25:56.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:56.900+0000 7f1c7f292640 1 -- 172.21.15.87:0/2703524698 wait complete. 2024-04-26T02:25:56.921 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-26T02:25:57.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f097f34b640 1 Processor -- start 2024-04-26T02:25:57.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f097f34b640 1 -- start start 2024-04-26T02:25:57.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f097f34b640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09780729f0 con 0x7f097813f7f0 2024-04-26T02:25:57.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f097f34b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0978144920 con 0x7f09780a8fc0 2024-04-26T02:25:57.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f097f34b640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09780ab000 con 0x7f0978072610 2024-04-26T02:25:57.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f097d8e8640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f097813f7f0 0x7f097813fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:60628/0 (socket says 172.21.15.87:60628) 2024-04-26T02:25:57.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f097d0e7640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f0978072610 0x7f0978140490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40484/0 (socket says 172.21.15.87:40484) 2024-04-26T02:25:57.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f097d8e8640 1 -- 172.21.15.87:0/3277797783 learned_addr learned my addr 172.21.15.87:0/3277797783 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:57.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f096ffff640 1 -- 172.21.15.87:0/3277797783 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2559105615 0 0) 0x7f09700014a0 con 0x7f097813f7f0 2024-04-26T02:25:57.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f096ffff640 1 -- 172.21.15.87:0/3277797783 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0964003530 con 0x7f097813f7f0 2024-04-26T02:25:57.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f096ffff640 1 -- 172.21.15.87:0/3277797783 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1574035486 0 0) 0x7f09780ab000 con 0x7f0978072610 2024-04-26T02:25:57.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f096ffff640 1 -- 172.21.15.87:0/3277797783 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09700014a0 con 0x7f0978072610 2024-04-26T02:25:57.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f096ffff640 1 -- 172.21.15.87:0/3277797783 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 578468300 0 0) 0x7f0964003530 con 0x7f097813f7f0 2024-04-26T02:25:57.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f096ffff640 1 -- 172.21.15.87:0/3277797783 >> v1:172.21.15.87:6790/0 conn(0x7f0978072610 legacy=0x7f0978140490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:57.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f096ffff640 1 -- 172.21.15.87:0/3277797783 >> v1:172.21.15.122:6789/0 conn(0x7f09780a8fc0 legacy=0x7f0978144210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:57.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f096ffff640 1 -- 172.21.15.87:0/3277797783 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0978143a10 con 0x7f097813f7f0 2024-04-26T02:25:57.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f096ffff640 1 -- 172.21.15.87:0/3277797783 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f0970001b70 con 0x7f097813f7f0 2024-04-26T02:25:57.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f096ffff640 1 -- 172.21.15.87:0/3277797783 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0970001da0 con 0x7f097813f7f0 2024-04-26T02:25:57.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f096ffff640 1 -- 172.21.15.87:0/3277797783 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f0970004c30 con 0x7f097813f7f0 2024-04-26T02:25:57.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.056+0000 7f097f34b640 1 -- 172.21.15.87:0/3277797783 >> v1:172.21.15.87:6789/0 conn(0x7f097813f7f0 legacy=0x7f097813fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:57.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f097f34b640 1 -- 172.21.15.87:0/3277797783 shutdown_connections 2024-04-26T02:25:57.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f097f34b640 1 -- 172.21.15.87:0/3277797783 >> 172.21.15.87:0/3277797783 conn(0x7f097806c490 msgr2=0x7f097806c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:57.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f097f34b640 1 -- 172.21.15.87:0/3277797783 shutdown_connections 2024-04-26T02:25:57.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f097f34b640 1 -- 172.21.15.87:0/3277797783 wait complete. 2024-04-26T02:25:57.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f097f34b640 1 Processor -- start 2024-04-26T02:25:57.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f097f34b640 1 -- start start 2024-04-26T02:25:57.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f097f34b640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f097812de80 con 0x7f0978072610 2024-04-26T02:25:57.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f097f34b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f097812e160 con 0x7f097813f7f0 2024-04-26T02:25:57.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f097f34b640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f097812fba0 con 0x7f09780a8fc0 2024-04-26T02:25:57.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f097d0e7640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f0978072610 0x7f097812e5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:60634/0 (socket says 172.21.15.87:60634) 2024-04-26T02:25:57.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f097d0e7640 1 -- 172.21.15.87:0/2915912114 learned_addr learned my addr 172.21.15.87:0/2915912114 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:57.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3426520027 0 0) 0x7f097812fba0 con 0x7f09780a8fc0 2024-04-26T02:25:57.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f095c003530 con 0x7f09780a8fc0 2024-04-26T02:25:57.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3509833474 0 0) 0x7f097812de80 con 0x7f0978072610 2024-04-26T02:25:57.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f097812fba0 con 0x7f0978072610 2024-04-26T02:25:57.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2638776724 0 0) 0x7f097812e160 con 0x7f097813f7f0 2024-04-26T02:25:57.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f097812de80 con 0x7f097813f7f0 2024-04-26T02:25:57.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2969024489 0 0) 0x7f097812fba0 con 0x7f0978072610 2024-04-26T02:25:57.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f097812e160 con 0x7f0978072610 2024-04-26T02:25:57.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2199474515 0 0) 0x7f095c003530 con 0x7f09780a8fc0 2024-04-26T02:25:57.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f097812fba0 con 0x7f09780a8fc0 2024-04-26T02:25:57.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f0974001840 con 0x7f0978072610 2024-04-26T02:25:57.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f0968001840 con 0x7f09780a8fc0 2024-04-26T02:25:57.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2045060781 0 0) 0x7f097812fba0 con 0x7f09780a8fc0 2024-04-26T02:25:57.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 >> v1:172.21.15.122:6789/0 conn(0x7f097813f7f0 legacy=0x7f09781dfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:57.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 >> v1:172.21.15.87:6789/0 conn(0x7f0978072610 legacy=0x7f097812e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:57.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097812f750 con 0x7f09780a8fc0 2024-04-26T02:25:57.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0968003620 con 0x7f09780a8fc0 2024-04-26T02:25:57.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f097f34b640 1 -- 172.21.15.87:0/2915912114 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f097812cec0 con 0x7f09780a8fc0 2024-04-26T02:25:57.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f09680037d0 con 0x7f09780a8fc0 2024-04-26T02:25:57.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.060+0000 7f097f34b640 1 -- 172.21.15.87:0/2915912114 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0978140b50 con 0x7f09780a8fc0 2024-04-26T02:25:57.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.064+0000 7f097f34b640 1 -- 172.21.15.87:0/2915912114 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f097804fd20 con 0x7f09780a8fc0 2024-04-26T02:25:57.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.064+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f0968019040 con 0x7f09780a8fc0 2024-04-26T02:25:57.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.064+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f096807ff70 con 0x7f09780a8fc0 2024-04-26T02:25:57.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.068+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f0968050de0 con 0x7f09780a8fc0 2024-04-26T02:25:57.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.280+0000 7f097f34b640 1 -- 172.21.15.87:0/2915912114 --> v1:172.21.15.122:6816/416193625 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f0978130150 con 0x7f095c0684a0 2024-04-26T02:25:57.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.280+0000 7f096dffb640 1 -- 172.21.15.87:0/2915912114 <== mgr.4108 v1:172.21.15.122:6816/416193625 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (unknown 1239724482 0 0) 0x7f0978130150 con 0x7f095c0684a0 2024-04-26T02:25:57.283 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 7 to deep-scrub 2024-04-26T02:25:57.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.284+0000 7f097f34b640 1 -- 172.21.15.87:0/2915912114 >> v1:172.21.15.122:6816/416193625 conn(0x7f095c0684a0 legacy=0x7f095c06a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:57.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.284+0000 7f097f34b640 1 -- 172.21.15.87:0/2915912114 >> v1:172.21.15.87:6790/0 conn(0x7f09780a8fc0 legacy=0x7f097812ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:57.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.284+0000 7f097f34b640 1 -- 172.21.15.87:0/2915912114 shutdown_connections 2024-04-26T02:25:57.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.284+0000 7f097f34b640 1 -- 172.21.15.87:0/2915912114 >> 172.21.15.87:0/2915912114 conn(0x7f097806c490 msgr2=0x7f097806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:57.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.284+0000 7f097f34b640 1 -- 172.21.15.87:0/2915912114 shutdown_connections 2024-04-26T02:25:57.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.284+0000 7f097f34b640 1 -- 172.21.15.87:0/2915912114 wait complete. 2024-04-26T02:25:57.306 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:25:57.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63d749a640 1 Processor -- start 2024-04-26T02:25:57.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63d749a640 1 -- start start 2024-04-26T02:25:57.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63d749a640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63d0067490 con 0x7f63d0146470 2024-04-26T02:25:57.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63d749a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63d00ad110 con 0x7f63d0071c70 2024-04-26T02:25:57.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63d749a640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63d0072cf0 con 0x7f63d00670b0 2024-04-26T02:25:57.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63d5236640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f63d00670b0 0x7f63d0141a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40506/0 (socket says 172.21.15.87:40506) 2024-04-26T02:25:57.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63d5236640 1 -- 172.21.15.87:0/3223014313 learned_addr learned my addr 172.21.15.87:0/3223014313 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:57.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63c7fff640 1 -- 172.21.15.87:0/3223014313 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3470669142 0 0) 0x7f63c80014a0 con 0x7f63d0146470 2024-04-26T02:25:57.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63c7fff640 1 -- 172.21.15.87:0/3223014313 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63bc003530 con 0x7f63d0146470 2024-04-26T02:25:57.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63c7fff640 1 -- 172.21.15.87:0/3223014313 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4193152258 0 0) 0x7f63d0072cf0 con 0x7f63d00670b0 2024-04-26T02:25:57.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63c7fff640 1 -- 172.21.15.87:0/3223014313 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63c80014a0 con 0x7f63d00670b0 2024-04-26T02:25:57.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63c7fff640 1 -- 172.21.15.87:0/3223014313 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1579240292 0 0) 0x7f63c80014a0 con 0x7f63d00670b0 2024-04-26T02:25:57.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63c7fff640 1 -- 172.21.15.87:0/3223014313 >> v1:172.21.15.122:6789/0 conn(0x7f63d0071c70 legacy=0x7f63d0145d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:57.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63c7fff640 1 -- 172.21.15.87:0/3223014313 >> v1:172.21.15.87:6789/0 conn(0x7f63d0146470 legacy=0x7f63d014dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:57.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63c7fff640 1 -- 172.21.15.87:0/3223014313 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63d01468c0 con 0x7f63d00670b0 2024-04-26T02:25:57.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63c7fff640 1 -- 172.21.15.87:0/3223014313 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f63cc001530 con 0x7f63d00670b0 2024-04-26T02:25:57.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63c7fff640 1 -- 172.21.15.87:0/3223014313 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f63cc0017a0 con 0x7f63d00670b0 2024-04-26T02:25:57.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63c7fff640 1 -- 172.21.15.87:0/3223014313 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f63cc001bb0 con 0x7f63d00670b0 2024-04-26T02:25:57.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63d749a640 1 -- 172.21.15.87:0/3223014313 >> v1:172.21.15.87:6790/0 conn(0x7f63d00670b0 legacy=0x7f63d0141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:57.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63d749a640 1 -- 172.21.15.87:0/3223014313 shutdown_connections 2024-04-26T02:25:57.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63d749a640 1 -- 172.21.15.87:0/3223014313 >> 172.21.15.87:0/3223014313 conn(0x7f63d006b7b0 msgr2=0x7f63d006bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:57.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63d749a640 1 -- 172.21.15.87:0/3223014313 shutdown_connections 2024-04-26T02:25:57.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.424+0000 7f63d749a640 1 -- 172.21.15.87:0/3223014313 wait complete. 2024-04-26T02:25:57.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63d749a640 1 Processor -- start 2024-04-26T02:25:57.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63d749a640 1 -- start start 2024-04-26T02:25:57.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63d749a640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63d011db40 con 0x7f63d0071c70 2024-04-26T02:25:57.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63d749a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63d011c4b0 con 0x7f63d00670b0 2024-04-26T02:25:57.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63d749a640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63d011c0d0 con 0x7f63d0146470 2024-04-26T02:25:57.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63d4a35640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f63d0071c70 0x7f63d01db0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:60652/0 (socket says 172.21.15.87:60652) 2024-04-26T02:25:57.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63d4a35640 1 -- 172.21.15.87:0/2462632691 learned_addr learned my addr 172.21.15.87:0/2462632691 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:25:57.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63d5a37640 1 --1- 172.21.15.87:0/2462632691 >> v1:172.21.15.87:6790/0 conn(0x7f63d0146470 0x7f63d01db7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40512/0 (socket says 172.21.15.87:40512) 2024-04-26T02:25:57.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3692232767 0 0) 0x7f63d011c0d0 con 0x7f63d0146470 2024-04-26T02:25:57.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63b4003530 con 0x7f63d0146470 2024-04-26T02:25:57.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1086714811 0 0) 0x7f63d011db40 con 0x7f63d0071c70 2024-04-26T02:25:57.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63d011c0d0 con 0x7f63d0071c70 2024-04-26T02:25:57.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1866480282 0 0) 0x7f63d011c4b0 con 0x7f63d00670b0 2024-04-26T02:25:57.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63d011db40 con 0x7f63d00670b0 2024-04-26T02:25:57.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3544440480 0 0) 0x7f63b4003530 con 0x7f63d0146470 2024-04-26T02:25:57.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63d011c4b0 con 0x7f63d0146470 2024-04-26T02:25:57.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f63c8001d90 con 0x7f63d0146470 2024-04-26T02:25:57.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 504122590 0 0) 0x7f63d011c0d0 con 0x7f63d0071c70 2024-04-26T02:25:57.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63b4003530 con 0x7f63d0071c70 2024-04-26T02:25:57.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f63c0001900 con 0x7f63d0071c70 2024-04-26T02:25:57.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1433524451 0 0) 0x7f63d011c4b0 con 0x7f63d0146470 2024-04-26T02:25:57.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 >> v1:172.21.15.122:6789/0 conn(0x7f63d00670b0 legacy=0x7f63d01da1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:57.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 >> v1:172.21.15.87:6789/0 conn(0x7f63d0071c70 legacy=0x7f63d01db0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:57.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63d01da8d0 con 0x7f63d0146470 2024-04-26T02:25:57.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63d749a640 1 -- 172.21.15.87:0/2462632691 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f63d01dabe0 con 0x7f63d0146470 2024-04-26T02:25:57.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63d749a640 1 -- 172.21.15.87:0/2462632691 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f63d00b04c0 con 0x7f63d0146470 2024-04-26T02:25:57.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f63c80020b0 con 0x7f63d0146470 2024-04-26T02:25:57.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f63c8003b20 con 0x7f63d0146470 2024-04-26T02:25:57.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63d749a640 1 -- 172.21.15.87:0/2462632691 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63b0005090 con 0x7f63d0146470 2024-04-26T02:25:57.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.428+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f63c8003cc0 con 0x7f63d0146470 2024-04-26T02:25:57.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.432+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f63c8081370 con 0x7f63d0146470 2024-04-26T02:25:57.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.436+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f63d0067490 con 0x7f63d0146470 2024-04-26T02:25:57.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.604+0000 7f63d749a640 1 -- 172.21.15.87:0/2462632691 --> v1:172.21.15.122:6816/416193625 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f63b0000ba0 con 0x7f63b4068450 2024-04-26T02:25:57.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.612+0000 7f63c5ffb640 1 -- 172.21.15.87:0/2462632691 <== mgr.4108 v1:172.21.15.122:6816/416193625 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+117477 (unknown 2965378022 0 2412047257) 0x7f63b0000ba0 con 0x7f63b4068450 2024-04-26T02:25:57.616 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:25:57.616 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-26T02:25:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.616+0000 7f63d749a640 1 -- 172.21.15.87:0/2462632691 >> v1:172.21.15.122:6816/416193625 conn(0x7f63b4068450 legacy=0x7f63b406a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.616+0000 7f63d749a640 1 -- 172.21.15.87:0/2462632691 >> v1:172.21.15.87:6790/0 conn(0x7f63d0146470 legacy=0x7f63d01db7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:25:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.616+0000 7f63d749a640 1 -- 172.21.15.87:0/2462632691 shutdown_connections 2024-04-26T02:25:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.616+0000 7f63d749a640 1 -- 172.21.15.87:0/2462632691 >> 172.21.15.87:0/2462632691 conn(0x7f63d006b7b0 msgr2=0x7f63d00b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:25:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.616+0000 7f63d749a640 1 -- 172.21.15.87:0/2462632691 shutdown_connections 2024-04-26T02:25:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:25:57.616+0000 7f63d749a640 1 -- 172.21.15.87:0/2462632691 wait complete. 2024-04-26T02:25:57.636 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-04-26T02:25:57.417849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3568980760,"num_objects":879,"num_object_clones":0,"num_object_copies":1758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":879,"num_whiteouts":0,"num_read":2675,"num_read_kb":2675,"num_write":159887,"num_write_kb":5638997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161446,"ondisk_log_size":161446,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":9156476,"kb_used_data":6640416,"kb_used_omap":12,"kb_used_meta":2515955,"kb_avail":745818244,"statfs":{"total":773094113280,"available":763717881856,"internally_reserved":0,"allocated":6799785984,"data_stored":6837114624,"data_compressed":194304976,"data_compressed_allocated":202137600,"data_compressed_original":244916224,"omap_allocated":12751,"internal_metadata":2576338481},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8720,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":66,"num_read_kb":71,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003033"},"pg_stats":[{"pgid":"3.1f","version":"22'2749","reported_seq":2771,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.581677+0000","last_change":"2024-04-26T02:21:37.492661+0000","last_active":"2024-04-26T02:25:50.581677+0000","last_peered":"2024-04-26T02:25:50.581677+0000","last_clean":"2024-04-26T02:25:50.581677+0000","last_became_active":"2024-04-26T02:21:37.491078+0000","last_became_peered":"2024-04-26T02:21:37.491078+0000","last_unstale":"2024-04-26T02:25:50.581677+0000","last_undegraded":"2024-04-26T02:25:50.581677+0000","last_fullsized":"2024-04-26T02:25:50.581677+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":2749,"log_dups_size":0,"ondisk_log_size":2749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":71,"num_write":2855,"num_write_kb":123586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"24'6577","reported_seq":6595,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.579838+0000","last_change":"2024-04-26T02:21:37.485093+0000","last_active":"2024-04-26T02:25:50.579838+0000","last_peered":"2024-04-26T02:25:50.579838+0000","last_clean":"2024-04-26T02:25:50.579838+0000","last_became_active":"2024-04-26T02:21:37.484287+0000","last_became_peered":"2024-04-26T02:21:37.484287+0000","last_unstale":"2024-04-26T02:25:50.579838+0000","last_undegraded":"2024-04-26T02:25:50.579838+0000","last_fullsized":"2024-04-26T02:25:50.579838+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":6577,"log_dups_size":0,"ondisk_log_size":6577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:56:08.755136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6549,"num_write_kb":206212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"24'5725","reported_seq":5762,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.672459+0000","last_change":"2024-04-26T02:21:37.498519+0000","last_active":"2024-04-26T02:25:50.672459+0000","last_peered":"2024-04-26T02:25:50.672459+0000","last_clean":"2024-04-26T02:25:50.672459+0000","last_became_active":"2024-04-26T02:21:37.497700+0000","last_became_peered":"2024-04-26T02:21:37.497700+0000","last_unstale":"2024-04-26T02:25:50.672459+0000","last_undegraded":"2024-04-26T02:25:50.672459+0000","last_fullsized":"2024-04-26T02:25:50.672459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":5725,"log_dups_size":0,"ondisk_log_size":5725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:02:58.912915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":76,"num_read_kb":124,"num_write":5736,"num_write_kb":155565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"22'4059","reported_seq":4079,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.676918+0000","last_change":"2024-04-26T02:21:37.497650+0000","last_active":"2024-04-26T02:25:50.676918+0000","last_peered":"2024-04-26T02:25:50.676918+0000","last_clean":"2024-04-26T02:25:50.676918+0000","last_became_active":"2024-04-26T02:21:37.496863+0000","last_became_peered":"2024-04-26T02:21:37.496863+0000","last_unstale":"2024-04-26T02:25:50.676918+0000","last_undegraded":"2024-04-26T02:25:50.676918+0000","last_fullsized":"2024-04-26T02:25:50.676918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4059,"log_dups_size":0,"ondisk_log_size":4059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:57:07.038484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":4059,"num_write_kb":185415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"24'3950","reported_seq":3977,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.687544+0000","last_change":"2024-04-26T02:21:37.491890+0000","last_active":"2024-04-26T02:25:50.687544+0000","last_peered":"2024-04-26T02:25:50.687544+0000","last_clean":"2024-04-26T02:25:50.687544+0000","last_became_active":"2024-04-26T02:21:37.490474+0000","last_became_peered":"2024-04-26T02:21:37.490474+0000","last_unstale":"2024-04-26T02:25:50.687544+0000","last_undegraded":"2024-04-26T02:25:50.687544+0000","last_fullsized":"2024-04-26T02:25:50.687544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3950,"log_dups_size":0,"ondisk_log_size":3950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120324096,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":34,"num_read_kb":151,"num_write":3926,"num_write_kb":190983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"24'6082","reported_seq":6105,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.596635+0000","last_change":"2024-04-26T02:22:35.114133+0000","last_active":"2024-04-26T02:25:50.596635+0000","last_peered":"2024-04-26T02:25:50.596635+0000","last_clean":"2024-04-26T02:25:50.596635+0000","last_became_active":"2024-04-26T02:21:37.485110+0000","last_became_peered":"2024-04-26T02:21:37.485110+0000","last_unstale":"2024-04-26T02:25:50.596635+0000","last_undegraded":"2024-04-26T02:25:50.596635+0000","last_fullsized":"2024-04-26T02:25:50.596635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":6082,"log_dups_size":0,"ondisk_log_size":6082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:39:18.389527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":6064,"num_write_kb":187005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"24'5041","reported_seq":5078,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.677003+0000","last_change":"2024-04-26T02:21:37.489148+0000","last_active":"2024-04-26T02:25:50.677003+0000","last_peered":"2024-04-26T02:25:50.677003+0000","last_clean":"2024-04-26T02:25:50.677003+0000","last_became_active":"2024-04-26T02:21:37.488519+0000","last_became_peered":"2024-04-26T02:21:37.488519+0000","last_unstale":"2024-04-26T02:25:50.677003+0000","last_undegraded":"2024-04-26T02:25:50.677003+0000","last_fullsized":"2024-04-26T02:25:50.677003+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":5041,"log_dups_size":0,"ondisk_log_size":5041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117441024,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":5034,"num_write_kb":181716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"23'7153","reported_seq":7192,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.578039+0000","last_change":"2024-04-26T02:21:37.496383+0000","last_active":"2024-04-26T02:25:50.578039+0000","last_peered":"2024-04-26T02:25:50.578039+0000","last_clean":"2024-04-26T02:25:50.578039+0000","last_became_active":"2024-04-26T02:21:37.495694+0000","last_became_peered":"2024-04-26T02:21:37.495694+0000","last_unstale":"2024-04-26T02:25:50.578039+0000","last_undegraded":"2024-04-26T02:25:50.578039+0000","last_fullsized":"2024-04-26T02:25:50.578039+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":7153,"log_dups_size":0,"ondisk_log_size":7153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:27:30.478083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023936,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":80,"num_read_kb":65,"num_write":7184,"num_write_kb":197893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"23'8281","reported_seq":8307,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.582228+0000","last_change":"2024-04-26T02:21:37.490841+0000","last_active":"2024-04-26T02:25:50.582228+0000","last_peered":"2024-04-26T02:25:50.582228+0000","last_clean":"2024-04-26T02:25:50.582228+0000","last_became_active":"2024-04-26T02:21:37.490119+0000","last_became_peered":"2024-04-26T02:21:37.490119+0000","last_unstale":"2024-04-26T02:25:50.582228+0000","last_undegraded":"2024-04-26T02:25:50.582228+0000","last_fullsized":"2024-04-26T02:25:50.582228+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":8281,"log_dups_size":0,"ondisk_log_size":8281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137060352,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":8291,"num_write_kb":229215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"24'3757","reported_seq":3774,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.671575+0000","last_change":"2024-04-26T02:21:37.507074+0000","last_active":"2024-04-26T02:25:50.671575+0000","last_peered":"2024-04-26T02:25:50.671575+0000","last_clean":"2024-04-26T02:25:50.671575+0000","last_became_active":"2024-04-26T02:21:37.506309+0000","last_became_peered":"2024-04-26T02:21:37.506309+0000","last_unstale":"2024-04-26T02:25:50.671575+0000","last_undegraded":"2024-04-26T02:25:50.671575+0000","last_fullsized":"2024-04-26T02:25:50.671575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3757,"log_dups_size":0,"ondisk_log_size":3757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:49:23.247821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3724,"num_write_kb":119684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"23'7177","reported_seq":7194,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.611448+0000","last_change":"2024-04-26T02:21:37.487697+0000","last_active":"2024-04-26T02:25:50.611448+0000","last_peered":"2024-04-26T02:25:50.611448+0000","last_clean":"2024-04-26T02:25:50.611448+0000","last_became_active":"2024-04-26T02:21:37.486868+0000","last_became_peered":"2024-04-26T02:21:37.486868+0000","last_unstale":"2024-04-26T02:25:50.611448+0000","last_undegraded":"2024-04-26T02:25:50.611448+0000","last_fullsized":"2024-04-26T02:25:50.611448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":7177,"log_dups_size":0,"ondisk_log_size":7177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:26:13.460721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7176,"num_write_kb":198373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"24'4536","reported_seq":4553,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.686177+0000","last_change":"2024-04-26T02:21:37.488921+0000","last_active":"2024-04-26T02:25:50.686177+0000","last_peered":"2024-04-26T02:25:50.686177+0000","last_clean":"2024-04-26T02:25:50.686177+0000","last_became_active":"2024-04-26T02:21:37.487980+0000","last_became_peered":"2024-04-26T02:21:37.487980+0000","last_unstale":"2024-04-26T02:25:50.686177+0000","last_undegraded":"2024-04-26T02:25:50.686177+0000","last_fullsized":"2024-04-26T02:25:50.686177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4536,"log_dups_size":0,"ondisk_log_size":4536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:10.065671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4488,"num_write_kb":107395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.669027+0000","last_change":"2024-04-26T02:21:31.429340+0000","last_active":"2024-04-26T02:25:50.669027+0000","last_peered":"2024-04-26T02:25:50.669027+0000","last_clean":"2024-04-26T02:25:50.669027+0000","last_became_active":"2024-04-26T02:21:31.428657+0000","last_became_peered":"2024-04-26T02:21:31.428657+0000","last_unstale":"2024-04-26T02:25:50.669027+0000","last_undegraded":"2024-04-26T02:25:50.669027+0000","last_fullsized":"2024-04-26T02:25:50.669027+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:07:12.674824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"24'4911","reported_seq":4945,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:54.222593+0000","last_change":"2024-04-26T02:25:54.222676+0000","last_active":"2024-04-26T02:25:54.222593+0000","last_peered":"2024-04-26T02:25:54.222593+0000","last_clean":"2024-04-26T02:25:54.222593+0000","last_became_active":"2024-04-26T02:21:37.481620+0000","last_became_peered":"2024-04-26T02:21:37.481620+0000","last_unstale":"2024-04-26T02:25:54.222593+0000","last_undegraded":"2024-04-26T02:25:54.222593+0000","last_fullsized":"2024-04-26T02:25:54.222593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4911","last_scrub_stamp":"2024-04-26T02:25:54.222501+0000","last_deep_scrub":"24'4911","last_deep_scrub_stamp":"2024-04-26T02:25:54.222501+0000","last_clean_scrub_stamp":"2024-04-26T02:25:54.222501+0000","objects_scrubbed":24,"log_size":4911,"log_dups_size":0,"ondisk_log_size":4911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:18.375457+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96600064,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":4875,"num_write_kb":131621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.583217+0000","last_change":"2024-04-26T02:21:31.432330+0000","last_active":"2024-04-26T02:25:50.583217+0000","last_peered":"2024-04-26T02:25:50.583217+0000","last_clean":"2024-04-26T02:25:50.583217+0000","last_became_active":"2024-04-26T02:21:31.431958+0000","last_became_peered":"2024-04-26T02:21:31.431958+0000","last_unstale":"2024-04-26T02:25:50.583217+0000","last_undegraded":"2024-04-26T02:25:50.583217+0000","last_fullsized":"2024-04-26T02:25:50.583217+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"24'4224","reported_seq":4241,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.684736+0000","last_change":"2024-04-26T02:21:37.488836+0000","last_active":"2024-04-26T02:25:50.684736+0000","last_peered":"2024-04-26T02:25:50.684736+0000","last_clean":"2024-04-26T02:25:50.684736+0000","last_became_active":"2024-04-26T02:21:37.487980+0000","last_became_peered":"2024-04-26T02:21:37.487980+0000","last_unstale":"2024-04-26T02:25:50.684736+0000","last_undegraded":"2024-04-26T02:25:50.684736+0000","last_fullsized":"2024-04-26T02:25:50.684736+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4224,"log_dups_size":0,"ondisk_log_size":4224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:07:49.262710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4175,"num_write_kb":201729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.612220+0000","last_change":"2024-04-26T02:21:31.430988+0000","last_active":"2024-04-26T02:25:50.612220+0000","last_peered":"2024-04-26T02:25:50.612220+0000","last_clean":"2024-04-26T02:25:50.612220+0000","last_became_active":"2024-04-26T02:21:31.430227+0000","last_became_peered":"2024-04-26T02:21:31.430227+0000","last_unstale":"2024-04-26T02:25:50.612220+0000","last_undegraded":"2024-04-26T02:25:50.612220+0000","last_fullsized":"2024-04-26T02:25:50.612220+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:32:21.265195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'6121","reported_seq":6145,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.582780+0000","last_change":"2024-04-26T02:21:37.483474+0000","last_active":"2024-04-26T02:25:50.582780+0000","last_peered":"2024-04-26T02:25:50.582780+0000","last_clean":"2024-04-26T02:25:50.582780+0000","last_became_active":"2024-04-26T02:21:37.482894+0000","last_became_peered":"2024-04-26T02:21:37.482894+0000","last_unstale":"2024-04-26T02:25:50.582780+0000","last_undegraded":"2024-04-26T02:25:50.582780+0000","last_fullsized":"2024-04-26T02:25:50.582780+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":6121,"log_dups_size":0,"ondisk_log_size":6121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99354976,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":6134,"num_write_kb":151141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:54.876012+0000","last_change":"2024-04-26T02:25:54.876069+0000","last_active":"2024-04-26T02:25:54.876012+0000","last_peered":"2024-04-26T02:25:54.876012+0000","last_clean":"2024-04-26T02:25:54.876012+0000","last_became_active":"2024-04-26T02:21:31.431071+0000","last_became_peered":"2024-04-26T02:21:31.431071+0000","last_unstale":"2024-04-26T02:25:54.876012+0000","last_undegraded":"2024-04-26T02:25:54.876012+0000","last_fullsized":"2024-04-26T02:25:54.876012+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:25:54.875820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:25:54.875820+0000","last_clean_scrub_stamp":"2024-04-26T02:25:54.875820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:17:31.825034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"22'4180","reported_seq":4227,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.673178+0000","last_change":"2024-04-26T02:21:37.490970+0000","last_active":"2024-04-26T02:25:50.673178+0000","last_peered":"2024-04-26T02:25:50.673178+0000","last_clean":"2024-04-26T02:25:50.673178+0000","last_became_active":"2024-04-26T02:21:37.490105+0000","last_became_peered":"2024-04-26T02:21:37.490105+0000","last_unstale":"2024-04-26T02:25:50.673178+0000","last_undegraded":"2024-04-26T02:25:50.673178+0000","last_fullsized":"2024-04-26T02:25:50.673178+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4180,"log_dups_size":0,"ondisk_log_size":4180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:17.335251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121896960,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":122,"num_read_kb":98,"num_write":4227,"num_write_kb":167240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"22'2962","reported_seq":2982,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.596895+0000","last_change":"2024-04-26T02:21:37.494426+0000","last_active":"2024-04-26T02:25:50.596895+0000","last_peered":"2024-04-26T02:25:50.596895+0000","last_clean":"2024-04-26T02:25:50.596895+0000","last_became_active":"2024-04-26T02:21:37.491249+0000","last_became_peered":"2024-04-26T02:21:37.491249+0000","last_unstale":"2024-04-26T02:25:50.596895+0000","last_undegraded":"2024-04-26T02:25:50.596895+0000","last_fullsized":"2024-04-26T02:25:50.596895+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:22:39.029300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74842112,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":7,"num_write":2966,"num_write_kb":163586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.671639+0000","last_change":"2024-04-26T02:21:32.425721+0000","last_active":"2024-04-26T02:25:50.671639+0000","last_peered":"2024-04-26T02:25:50.671639+0000","last_clean":"2024-04-26T02:25:50.671639+0000","last_became_active":"2024-04-26T02:21:32.425189+0000","last_became_peered":"2024-04-26T02:21:32.425189+0000","last_unstale":"2024-04-26T02:25:50.671639+0000","last_undegraded":"2024-04-26T02:25:50.671639+0000","last_fullsized":"2024-04-26T02:25:50.671639+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:56:47.156837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"24'3483","reported_seq":3508,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.678939+0000","last_change":"2024-04-26T02:21:37.486201+0000","last_active":"2024-04-26T02:25:50.678939+0000","last_peered":"2024-04-26T02:25:50.678939+0000","last_clean":"2024-04-26T02:25:50.678939+0000","last_became_active":"2024-04-26T02:21:37.485432+0000","last_became_peered":"2024-04-26T02:21:37.485432+0000","last_unstale":"2024-04-26T02:25:50.678939+0000","last_undegraded":"2024-04-26T02:25:50.678939+0000","last_fullsized":"2024-04-26T02:25:50.678939+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3483,"log_dups_size":0,"ondisk_log_size":3483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86706520,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":3452,"num_write_kb":123177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.585595+0000","last_change":"2024-04-26T02:21:31.436585+0000","last_active":"2024-04-26T02:25:50.585595+0000","last_peered":"2024-04-26T02:25:50.585595+0000","last_clean":"2024-04-26T02:25:50.585595+0000","last_became_active":"2024-04-26T02:21:31.436226+0000","last_became_peered":"2024-04-26T02:21:31.436226+0000","last_unstale":"2024-04-26T02:25:50.585595+0000","last_undegraded":"2024-04-26T02:25:50.585595+0000","last_fullsized":"2024-04-26T02:25:50.585595+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:21:29.470825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"24'4456","reported_seq":4476,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.683694+0000","last_change":"2024-04-26T02:21:37.494384+0000","last_active":"2024-04-26T02:25:50.683694+0000","last_peered":"2024-04-26T02:25:50.683694+0000","last_clean":"2024-04-26T02:25:50.683694+0000","last_became_active":"2024-04-26T02:21:37.493922+0000","last_became_peered":"2024-04-26T02:21:37.493922+0000","last_unstale":"2024-04-26T02:25:50.683694+0000","last_undegraded":"2024-04-26T02:25:50.683694+0000","last_fullsized":"2024-04-26T02:25:50.683694+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4456,"log_dups_size":0,"ondisk_log_size":4456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113312264,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":4436,"num_write_kb":167962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.611264+0000","last_change":"2024-04-26T02:21:31.433313+0000","last_active":"2024-04-26T02:25:50.611264+0000","last_peered":"2024-04-26T02:25:50.611264+0000","last_clean":"2024-04-26T02:25:50.611264+0000","last_became_active":"2024-04-26T02:21:31.432395+0000","last_became_peered":"2024-04-26T02:21:31.432395+0000","last_unstale":"2024-04-26T02:25:50.611264+0000","last_undegraded":"2024-04-26T02:25:50.611264+0000","last_fullsized":"2024-04-26T02:25:50.611264+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:57:44.139176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"23'4930","reported_seq":4961,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.578656+0000","last_change":"2024-04-26T02:21:37.485786+0000","last_active":"2024-04-26T02:25:50.578656+0000","last_peered":"2024-04-26T02:25:50.578656+0000","last_clean":"2024-04-26T02:25:50.578656+0000","last_became_active":"2024-04-26T02:21:37.484172+0000","last_became_peered":"2024-04-26T02:21:37.484172+0000","last_unstale":"2024-04-26T02:25:50.578656+0000","last_undegraded":"2024-04-26T02:25:50.578656+0000","last_fullsized":"2024-04-26T02:25:50.578656+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:40:57.948908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91030032,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":49,"num_read_kb":40,"num_write":4960,"num_write_kb":192278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.671915+0000","last_change":"2024-04-26T02:21:33.341715+0000","last_active":"2024-04-26T02:25:50.671915+0000","last_peered":"2024-04-26T02:25:50.671915+0000","last_clean":"2024-04-26T02:25:50.671915+0000","last_became_active":"2024-04-26T02:21:33.340937+0000","last_became_peered":"2024-04-26T02:21:33.340937+0000","last_unstale":"2024-04-26T02:25:50.671915+0000","last_undegraded":"2024-04-26T02:25:50.671915+0000","last_fullsized":"2024-04-26T02:25:50.671915+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:30.392677+0000","last_clean_scrub_stamp":"2024-04-26T02:21:30.392677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:03:17.530982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":180,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.611288+0000","last_change":"2024-04-26T02:21:29.434185+0000","last_active":"2024-04-26T02:25:50.611288+0000","last_peered":"2024-04-26T02:25:50.611288+0000","last_clean":"2024-04-26T02:25:50.611288+0000","last_became_active":"2024-04-26T02:21:29.432363+0000","last_became_peered":"2024-04-26T02:21:29.432363+0000","last_unstale":"2024-04-26T02:25:50.611288+0000","last_undegraded":"2024-04-26T02:25:50.611288+0000","last_fullsized":"2024-04-26T02:25:50.611288+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:28.375870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:28.375870+0000","last_clean_scrub_stamp":"2024-04-26T02:21:28.375870+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:18:11.058445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"24'4894","reported_seq":4911,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.682487+0000","last_change":"2024-04-26T02:21:37.488171+0000","last_active":"2024-04-26T02:25:50.682487+0000","last_peered":"2024-04-26T02:25:50.682487+0000","last_clean":"2024-04-26T02:25:50.682487+0000","last_became_active":"2024-04-26T02:21:37.486673+0000","last_became_peered":"2024-04-26T02:21:37.486673+0000","last_unstale":"2024-04-26T02:25:50.682487+0000","last_undegraded":"2024-04-26T02:25:50.682487+0000","last_fullsized":"2024-04-26T02:25:50.682487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:29:05.728663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4819,"num_write_kb":226567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"24'9655","reported_seq":9782,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.724428+0000","last_change":"2024-04-26T02:21:37.476830+0000","last_active":"2024-04-26T02:25:50.724428+0000","last_peered":"2024-04-26T02:25:50.724428+0000","last_clean":"2024-04-26T02:25:50.724428+0000","last_became_active":"2024-04-26T02:21:37.476072+0000","last_became_peered":"2024-04-26T02:21:37.476072+0000","last_unstale":"2024-04-26T02:25:50.724428+0000","last_undegraded":"2024-04-26T02:25:50.724428+0000","last_fullsized":"2024-04-26T02:25:50.724428+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":9655,"log_dups_size":0,"ondisk_log_size":9655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:26:14.037852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138481144,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":373,"num_read_kb":339,"num_write":7867,"num_write_kb":214107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"24'5769","reported_seq":5895,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.666709+0000","last_change":"2024-04-26T02:21:37.490171+0000","last_active":"2024-04-26T02:25:50.666709+0000","last_peered":"2024-04-26T02:25:50.666709+0000","last_clean":"2024-04-26T02:25:50.666709+0000","last_became_active":"2024-04-26T02:21:37.488724+0000","last_became_peered":"2024-04-26T02:21:37.488724+0000","last_unstale":"2024-04-26T02:25:50.666709+0000","last_undegraded":"2024-04-26T02:25:50.666709+0000","last_fullsized":"2024-04-26T02:25:50.666709+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":5769,"log_dups_size":0,"ondisk_log_size":5769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:17.512625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134480400,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":132,"num_read_kb":124,"num_write":5774,"num_write_kb":201317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"22'3903","reported_seq":3920,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.579025+0000","last_change":"2024-04-26T02:21:37.492466+0000","last_active":"2024-04-26T02:25:50.579025+0000","last_peered":"2024-04-26T02:25:50.579025+0000","last_clean":"2024-04-26T02:25:50.579025+0000","last_became_active":"2024-04-26T02:21:37.491785+0000","last_became_peered":"2024-04-26T02:21:37.491785+0000","last_unstale":"2024-04-26T02:25:50.579025+0000","last_undegraded":"2024-04-26T02:25:50.579025+0000","last_fullsized":"2024-04-26T02:25:50.579025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3903,"log_dups_size":0,"ondisk_log_size":3903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:44.655312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88842240,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3901,"num_write_kb":140906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"24'3920","reported_seq":3939,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.687570+0000","last_change":"2024-04-26T02:21:37.493141+0000","last_active":"2024-04-26T02:25:50.687570+0000","last_peered":"2024-04-26T02:25:50.687570+0000","last_clean":"2024-04-26T02:25:50.687570+0000","last_became_active":"2024-04-26T02:21:37.490836+0000","last_became_peered":"2024-04-26T02:21:37.490836+0000","last_unstale":"2024-04-26T02:25:50.687570+0000","last_undegraded":"2024-04-26T02:25:50.687570+0000","last_fullsized":"2024-04-26T02:25:50.687570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3920,"log_dups_size":0,"ondisk_log_size":3920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:38:35.359899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100672016,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3880,"num_write_kb":189331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"24'3149","reported_seq":3474,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.611663+0000","last_change":"2024-04-26T02:21:37.501742+0000","last_active":"2024-04-26T02:25:50.611663+0000","last_peered":"2024-04-26T02:25:50.611663+0000","last_clean":"2024-04-26T02:25:50.611663+0000","last_became_active":"2024-04-26T02:21:37.500919+0000","last_became_peered":"2024-04-26T02:21:37.500919+0000","last_unstale":"2024-04-26T02:25:50.611663+0000","last_undegraded":"2024-04-26T02:25:50.611663+0000","last_fullsized":"2024-04-26T02:25:50.611663+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3149,"log_dups_size":0,"ondisk_log_size":3149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:15:44.491969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84148752,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1028,"num_read_kb":921,"num_write":3342,"num_write_kb":128651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"24'4862","reported_seq":4977,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.697203+0000","last_change":"2024-04-26T02:21:37.497393+0000","last_active":"2024-04-26T02:25:50.697203+0000","last_peered":"2024-04-26T02:25:50.697203+0000","last_clean":"2024-04-26T02:25:50.697203+0000","last_became_active":"2024-04-26T02:21:37.489359+0000","last_became_peered":"2024-04-26T02:21:37.489359+0000","last_unstale":"2024-04-26T02:25:50.697203+0000","last_undegraded":"2024-04-26T02:25:50.697203+0000","last_fullsized":"2024-04-26T02:25:50.697203+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4862,"log_dups_size":0,"ondisk_log_size":4862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:47.007579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130154496,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":323,"num_read_kb":315,"num_write":5125,"num_write_kb":178670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"24'3715","reported_seq":3732,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.685907+0000","last_change":"2024-04-26T02:21:37.489974+0000","last_active":"2024-04-26T02:25:50.685907+0000","last_peered":"2024-04-26T02:25:50.685907+0000","last_clean":"2024-04-26T02:25:50.685907+0000","last_became_active":"2024-04-26T02:21:37.488144+0000","last_became_peered":"2024-04-26T02:21:37.488144+0000","last_unstale":"2024-04-26T02:25:50.685907+0000","last_undegraded":"2024-04-26T02:25:50.685907+0000","last_fullsized":"2024-04-26T02:25:50.685907+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3715,"log_dups_size":0,"ondisk_log_size":3715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:53:06.375314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":168964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"24'3583","reported_seq":3603,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.686604+0000","last_change":"2024-04-26T02:21:37.495244+0000","last_active":"2024-04-26T02:25:50.686604+0000","last_peered":"2024-04-26T02:25:50.686604+0000","last_clean":"2024-04-26T02:25:50.686604+0000","last_became_active":"2024-04-26T02:21:37.494442+0000","last_became_peered":"2024-04-26T02:21:37.494442+0000","last_unstale":"2024-04-26T02:25:50.686604+0000","last_undegraded":"2024-04-26T02:25:50.686604+0000","last_fullsized":"2024-04-26T02:25:50.686604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":3583,"log_dups_size":0,"ondisk_log_size":3583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:36.918805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":3543,"num_write_kb":168836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"22'4310","reported_seq":4327,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.611250+0000","last_change":"2024-04-26T02:21:37.498351+0000","last_active":"2024-04-26T02:25:50.611250+0000","last_peered":"2024-04-26T02:25:50.611250+0000","last_clean":"2024-04-26T02:25:50.611250+0000","last_became_active":"2024-04-26T02:21:37.497599+0000","last_became_peered":"2024-04-26T02:21:37.497599+0000","last_unstale":"2024-04-26T02:25:50.611250+0000","last_undegraded":"2024-04-26T02:25:50.611250+0000","last_fullsized":"2024-04-26T02:25:50.611250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:05:22.895436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4304,"num_write_kb":209927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"24'4330","reported_seq":4354,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.685050+0000","last_change":"2024-04-26T02:21:37.487437+0000","last_active":"2024-04-26T02:25:50.685050+0000","last_peered":"2024-04-26T02:25:50.685050+0000","last_clean":"2024-04-26T02:25:50.685050+0000","last_became_active":"2024-04-26T02:21:37.486967+0000","last_became_peered":"2024-04-26T02:21:37.486967+0000","last_unstale":"2024-04-26T02:25:50.685050+0000","last_undegraded":"2024-04-26T02:25:50.685050+0000","last_fullsized":"2024-04-26T02:25:50.685050+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":4330,"log_dups_size":0,"ondisk_log_size":4330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127401984,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23,"num_read_kb":80,"num_write":4288,"num_write_kb":199362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"24'8885","reported_seq":8930,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:50.611653+0000","last_change":"2024-04-26T02:21:37.496905+0000","last_active":"2024-04-26T02:25:50.611653+0000","last_peered":"2024-04-26T02:25:50.611653+0000","last_clean":"2024-04-26T02:25:50.611653+0000","last_became_active":"2024-04-26T02:21:37.488117+0000","last_became_peered":"2024-04-26T02:21:37.488117+0000","last_unstale":"2024-04-26T02:25:50.611653+0000","last_undegraded":"2024-04-26T02:25:50.611653+0000","last_fullsized":"2024-04-26T02:25:50.611653+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:21:36.459647+0000","last_clean_scrub_stamp":"2024-04-26T02:21:36.459647+0000","objects_scrubbed":0,"log_size":8885,"log_dups_size":0,"ondisk_log_size":8885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:55:56.651035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128582144,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":108,"num_read_kb":91,"num_write":8909,"num_write_kb":228167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3568390392,"num_objects":877,"num_object_clones":0,"num_object_copies":1754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":877,"num_whiteouts":0,"num_read":2559,"num_read_kb":2575,"num_write":159728,"num_write_kb":5636581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6797877248,"data_stored":6835430520,"data_compressed":194293810,"data_compressed_allocated":202072064,"data_compressed_original":243892224,"omap_allocated":0,"internal_metadata":0},"log_size":161329,"ondisk_log_size":161329,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11166,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542200,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1412420,"kb_used_data":1038392,"kb_used_omap":1,"kb_used_meta":374014,"kb_avail":92959420,"statfs":{"total":96636764160,"available":95190446080,"internally_reserved":0,"allocated":1063313408,"data_stored":1066328537,"data_compressed":20319008,"data_compressed_allocated":21159936,"data_compressed_original":25022464,"omap_allocated":1590,"internal_metadata":382990794},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542201,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1148344,"kb_used_data":864556,"kb_used_omap":1,"kb_used_meta":283774,"kb_avail":93223496,"statfs":{"total":96636764160,"available":95460859904,"internally_reserved":0,"allocated":885305344,"data_stored":888268833,"data_compressed":21380520,"data_compressed_allocated":22167552,"data_compressed_original":25878528,"omap_allocated":1595,"internal_metadata":290585029},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542200,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1944996,"kb_used_data":1386328,"kb_used_omap":1,"kb_used_meta":558654,"kb_avail":92426844,"statfs":{"total":96636764160,"available":94645088256,"internally_reserved":0,"allocated":1419599872,"data_stored":1427699009,"data_compressed":38724772,"data_compressed_allocated":40361984,"data_compressed_original":49508352,"omap_allocated":1595,"internal_metadata":572062149},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542202,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1446360,"kb_used_data":1076492,"kb_used_omap":1,"kb_used_meta":369854,"kb_avail":92925480,"statfs":{"total":96636764160,"available":95155691520,"internally_reserved":0,"allocated":1102327808,"data_stored":1108250049,"data_compressed":31089397,"data_compressed_allocated":32296960,"data_compressed_original":39075840,"omap_allocated":1592,"internal_metadata":378730952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542200,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":885776,"kb_used_data":615880,"kb_used_omap":1,"kb_used_meta":269886,"kb_avail":93486064,"statfs":{"total":96636764160,"available":95729729536,"internally_reserved":0,"allocated":630661120,"data_stored":637301965,"data_compressed":27771469,"data_compressed_allocated":28930048,"data_compressed_original":36147200,"omap_allocated":1595,"internal_metadata":276363717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542200,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1387404,"kb_used_data":1018944,"kb_used_omap":1,"kb_used_meta":368446,"kb_avail":92984436,"statfs":{"total":96636764160,"available":95216062464,"internally_reserved":0,"allocated":1043398656,"data_stored":1049611477,"data_compressed":28674963,"data_compressed_allocated":29855744,"data_compressed_original":36777984,"omap_allocated":1595,"internal_metadata":377289157},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574905,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464504,"kb_used_data":343084,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93907336,"statfs":{"total":96636764160,"available":96161112064,"internally_reserved":0,"allocated":351318016,"data_stored":352441097,"data_compressed":9222947,"data_compressed_allocated":9637888,"data_compressed_original":11067392,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574905,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466672,"kb_used_data":296740,"kb_used_omap":1,"kb_used_meta":169918,"kb_avail":93905168,"statfs":{"total":96636764160,"available":96158892032,"internally_reserved":0,"allocated":303861760,"data_stored":307213657,"data_compressed":17121900,"data_compressed_allocated":17727488,"data_compressed_original":21438464,"omap_allocated":1595,"internal_metadata":173996485},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1043120128,"data_stored":1049004268,"data_compressed":28669380,"data_compressed_allocated":29822976,"data_compressed_original":36265984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":630435840,"data_stored":637223684,"data_compressed":27771469,"data_compressed_allocated":28930048,"data_compressed_original":36147200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303636480,"data_stored":307135376,"data_compressed":17121900,"data_compressed_allocated":17727488,"data_compressed_original":21438464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1102102528,"data_stored":1108171768,"data_compressed":31089397,"data_compressed_allocated":32296960,"data_compressed_original":39075840,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1419374592,"data_stored":1427620728,"data_compressed":38724772,"data_compressed_allocated":40361984,"data_compressed_original":49508352,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":885080064,"data_stored":888190552,"data_compressed":21380520,"data_compressed_allocated":22167552,"data_compressed_original":25878528,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":351092736,"data_stored":352362816,"data_compressed":9222947,"data_compressed_allocated":9637888,"data_compressed_original":11067392,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1063034880,"data_stored":1065721328,"data_compressed":20313425,"data_compressed_allocated":21127168,"data_compressed_original":24510464,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:25:57.641 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.641 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.641 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.641 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.641 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.641 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.641 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.642 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.642 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.642 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.642 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.642 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.642 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-26T02:21:30.392677+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=30, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.642 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-26T02:21:30.392677+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=30, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.642 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.642 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-26T02:21:30.392677+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=30, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.643 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.643 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.643 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.643 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-26T02:21:30.392677+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=30, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.643 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.643 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-26T02:21:30.392677+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=30, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.643 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.643 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-26T02:21:30.392677+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=30, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.643 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.644 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-26T02:21:30.392677+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=30, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.644 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-26T02:21:28.375870+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=28, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.644 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.644 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.644 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.644 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.644 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.644 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.644 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.644 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.645 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.645 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.645 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.645 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-26T02:21:36.459647+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=21, tm_sec=36, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=2, tm_min=25, tm_sec=52, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T02:25:57.645 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-26T02:26:17.646 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T02:26:17.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.776+0000 7f5c1571b640 1 Processor -- start 2024-04-26T02:26:17.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.776+0000 7f5c1571b640 1 -- start start 2024-04-26T02:26:17.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.776+0000 7f5c1571b640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c100aa220 con 0x7f5c1006f490 2024-04-26T02:26:17.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.776+0000 7f5c1571b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c1013f3f0 con 0x7f5c1006f0b0 2024-04-26T02:26:17.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.776+0000 7f5c1571b640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c1013f5c0 con 0x7f5c101402e0 2024-04-26T02:26:17.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.776+0000 7f5c0e7fc640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f5c1006f490 0x7f5c1013fbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:34700/0 (socket says 172.21.15.87:34700) 2024-04-26T02:26:17.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0e7fc640 1 -- 172.21.15.87:0/947451282 learned_addr learned my addr 172.21.15.87:0/947451282 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:26:17.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0dffb640 1 -- 172.21.15.87:0/947451282 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1882228436 0 0) 0x7f5c100aa220 con 0x7f5c1006f490 2024-04-26T02:26:17.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0dffb640 1 -- 172.21.15.87:0/947451282 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bf8005740 con 0x7f5c1006f490 2024-04-26T02:26:17.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0dffb640 1 -- 172.21.15.87:0/947451282 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3911388500 0 0) 0x7f5c080014a0 con 0x7f5c101402e0 2024-04-26T02:26:17.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0dffb640 1 -- 172.21.15.87:0/947451282 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c100aa220 con 0x7f5c101402e0 2024-04-26T02:26:17.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0dffb640 1 -- 172.21.15.87:0/947451282 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 933254820 0 0) 0x7f5bf8005740 con 0x7f5c1006f490 2024-04-26T02:26:17.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0dffb640 1 -- 172.21.15.87:0/947451282 >> v1:172.21.15.87:6790/0 conn(0x7f5c101402e0 legacy=0x7f5c10142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:26:17.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0dffb640 1 -- 172.21.15.87:0/947451282 >> v1:172.21.15.122:6789/0 conn(0x7f5c1006f0b0 legacy=0x7f5c100a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:26:17.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0dffb640 1 -- 172.21.15.87:0/947451282 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c10143460 con 0x7f5c1006f490 2024-04-26T02:26:17.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0dffb640 1 -- 172.21.15.87:0/947451282 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f5c04001620 con 0x7f5c1006f490 2024-04-26T02:26:17.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0dffb640 1 -- 172.21.15.87:0/947451282 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c04001880 con 0x7f5c1006f490 2024-04-26T02:26:17.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0dffb640 1 -- 172.21.15.87:0/947451282 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f5c04001c50 con 0x7f5c1006f490 2024-04-26T02:26:17.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c1571b640 1 -- 172.21.15.87:0/947451282 >> v1:172.21.15.87:6789/0 conn(0x7f5c1006f490 legacy=0x7f5c1013fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:26:17.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c1571b640 1 -- 172.21.15.87:0/947451282 shutdown_connections 2024-04-26T02:26:17.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c1571b640 1 -- 172.21.15.87:0/947451282 >> 172.21.15.87:0/947451282 conn(0x7f5c100b40d0 msgr2=0x7f5c100b44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:26:17.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c1571b640 1 -- 172.21.15.87:0/947451282 shutdown_connections 2024-04-26T02:26:17.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c1571b640 1 -- 172.21.15.87:0/947451282 wait complete. 2024-04-26T02:26:17.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c1571b640 1 Processor -- start 2024-04-26T02:26:17.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c1571b640 1 -- start start 2024-04-26T02:26:17.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c1571b640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c101341c0 con 0x7f5c1006f490 2024-04-26T02:26:17.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c1571b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c101d7490 con 0x7f5c101402e0 2024-04-26T02:26:17.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c1571b640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c10138cf0 con 0x7f5c1006f0b0 2024-04-26T02:26:17.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0effd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f5c1006f0b0 0x7f5c101d6670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:54358/0 (socket says 172.21.15.87:54358) 2024-04-26T02:26:17.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0e7fc640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f5c1006f490 0x7f5c101d6d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:34710/0 (socket says 172.21.15.87:34710) 2024-04-26T02:26:17.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0effd640 1 -- 172.21.15.87:0/905552354 learned_addr learned my addr 172.21.15.87:0/905552354 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:26:17.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0e7fc640 1 -- 172.21.15.87:0/905552354 >> v1:172.21.15.87:6789/0 conn(0x7f5c1006f490 legacy=0x7f5c101d6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2024-04-26T02:26:17.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0e7fc640 1 --1- 172.21.15.87:0/905552354 >> v1:172.21.15.87:6789/0 conn(0x7f5c1006f490 0x7f5c101d6d80 :-1 s=OPENED pgs=358 cs=1 l=1).write_message error sending 0x7f5c101341c0, (32) Broken pipe 2024-04-26T02:26:17.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0e7fc640 1 --1- 172.21.15.87:0/905552354 >> v1:172.21.15.87:6789/0 conn(0x7f5c1006f490 0x7f5c101d6d80 :-1 s=OPENED pgs=358 cs=1 l=1).write_event send msg failed 2024-04-26T02:26:17.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0e7fc640 1 --1- 172.21.15.87:0/905552354 >> v1:172.21.15.87:6789/0 conn(0x7f5c1006f490 0x7f5c101d6d80 :-1 s=OPENED pgs=358 cs=1 l=1).write_event send msg failed 2024-04-26T02:26:17.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5c0e7fc640 1 --1- 172.21.15.87:0/905552354 >> v1:172.21.15.87:6789/0 conn(0x7f5c1006f490 0x7f5c101d6d80 :-1 s=OPENED pgs=358 cs=1 l=1).fault on lossy channel, failing 2024-04-26T02:26:17.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4125268841 0 0) 0x7f5c10138cf0 con 0x7f5c1006f0b0 2024-04-26T02:26:17.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bfc003530 con 0x7f5c1006f0b0 2024-04-26T02:26:17.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 200062044 0 0) 0x7f5bfc003530 con 0x7f5c1006f0b0 2024-04-26T02:26:17.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c10138cf0 con 0x7f5c1006f0b0 2024-04-26T02:26:17.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.780+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f5c00001950 con 0x7f5c1006f0b0 2024-04-26T02:26:17.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.784+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3097590719 0 0) 0x7f5c101d7490 con 0x7f5c101402e0 2024-04-26T02:26:17.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.784+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bfc003530 con 0x7f5c101402e0 2024-04-26T02:26:17.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.784+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3304064918 0 0) 0x7f5c10138cf0 con 0x7f5c1006f0b0 2024-04-26T02:26:17.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.784+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 >> v1:172.21.15.122:6789/0 conn(0x7f5c101402e0 legacy=0x7f5c101ddbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:26:17.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.784+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 >> v1:172.21.15.87:6789/0 conn(0x7f5c1006f490 legacy=0x7f5c101d6d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T02:26:17.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.784+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c10136830 con 0x7f5c1006f0b0 2024-04-26T02:26:17.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.784+0000 7f5c1571b640 1 -- 172.21.15.87:0/905552354 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5c10133f30 con 0x7f5c1006f0b0 2024-04-26T02:26:17.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.784+0000 7f5c1571b640 1 -- 172.21.15.87:0/905552354 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5c10137370 con 0x7f5c1006f0b0 2024-04-26T02:26:17.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.784+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c00001d00 con 0x7f5c1006f0b0 2024-04-26T02:26:17.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.784+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f5c00003b70 con 0x7f5c1006f0b0 2024-04-26T02:26:17.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.784+0000 7f5c1571b640 1 -- 172.21.15.87:0/905552354 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bf0005090 con 0x7f5c1006f0b0 2024-04-26T02:26:17.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.784+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f5c00003d10 con 0x7f5c1006f0b0 2024-04-26T02:26:17.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.784+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f5c00081370 con 0x7f5c1006f0b0 2024-04-26T02:26:17.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.788+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f5c000521e0 con 0x7f5c1006f0b0 2024-04-26T02:26:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.944+0000 7f5c1571b640 1 -- 172.21.15.87:0/905552354 --> v1:172.21.15.122:6816/416193625 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5bf0000ba0 con 0x7f5bfc068430 2024-04-26T02:26:17.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.956+0000 7f5bf77fe640 1 -- 172.21.15.87:0/905552354 <== mgr.4108 v1:172.21.15.122:6816/416193625 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+118047 (unknown 2965378022 0 163799970) 0x7f5bf0000ba0 con 0x7f5bfc068430 2024-04-26T02:26:17.957 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-26T02:26:17.958 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-26T02:26:17.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.956+0000 7f5c1571b640 1 -- 172.21.15.87:0/905552354 >> v1:172.21.15.122:6816/416193625 conn(0x7f5bfc068430 legacy=0x7f5bfc06a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:26:17.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.956+0000 7f5c1571b640 1 -- 172.21.15.87:0/905552354 >> v1:172.21.15.87:6790/0 conn(0x7f5c1006f0b0 legacy=0x7f5c101d6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:26:17.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.956+0000 7f5c1571b640 1 -- 172.21.15.87:0/905552354 shutdown_connections 2024-04-26T02:26:17.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.956+0000 7f5c1571b640 1 -- 172.21.15.87:0/905552354 >> 172.21.15.87:0/905552354 conn(0x7f5c100b40d0 msgr2=0x7f5c101352a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:26:17.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.956+0000 7f5c1571b640 1 -- 172.21.15.87:0/905552354 shutdown_connections 2024-04-26T02:26:17.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:17.956+0000 7f5c1571b640 1 -- 172.21.15.87:0/905552354 wait complete. 2024-04-26T02:26:17.975 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-04-26T02:26:17.424392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3568980760,"num_objects":879,"num_object_clones":0,"num_object_copies":1758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":879,"num_whiteouts":0,"num_read":2675,"num_read_kb":2675,"num_write":159887,"num_write_kb":5638997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162299,"ondisk_log_size":162299,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":9143968,"kb_used_data":6640416,"kb_used_omap":12,"kb_used_meta":2503475,"kb_avail":745830752,"statfs":{"total":773094113280,"available":763730690048,"internally_reserved":0,"allocated":6799785984,"data_stored":6837114624,"data_compressed":194304976,"data_compressed_allocated":202137600,"data_compressed_original":244916224,"omap_allocated":12751,"internal_metadata":2563558961},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003991"},"pg_stats":[{"pgid":"3.1f","version":"24'2768","reported_seq":2798,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:58.368327+0000","last_change":"2024-04-26T02:25:58.368467+0000","last_active":"2024-04-26T02:25:58.368327+0000","last_peered":"2024-04-26T02:25:58.368327+0000","last_clean":"2024-04-26T02:25:58.368327+0000","last_became_active":"2024-04-26T02:21:37.491078+0000","last_became_peered":"2024-04-26T02:21:37.491078+0000","last_unstale":"2024-04-26T02:25:58.368327+0000","last_undegraded":"2024-04-26T02:25:58.368327+0000","last_fullsized":"2024-04-26T02:25:58.368327+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'2768","last_scrub_stamp":"2024-04-26T02:25:58.368165+0000","last_deep_scrub":"24'2768","last_deep_scrub_stamp":"2024-04-26T02:25:58.368165+0000","last_clean_scrub_stamp":"2024-04-26T02:25:58.368165+0000","objects_scrubbed":19,"log_size":2768,"log_dups_size":0,"ondisk_log_size":2768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:38:59.443707+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":71,"num_write":2855,"num_write_kb":123586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"24'6605","reported_seq":6633,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:56.540378+0000","last_change":"2024-04-26T02:25:56.540439+0000","last_active":"2024-04-26T02:25:56.540378+0000","last_peered":"2024-04-26T02:25:56.540378+0000","last_clean":"2024-04-26T02:25:56.540378+0000","last_became_active":"2024-04-26T02:21:37.484287+0000","last_became_peered":"2024-04-26T02:21:37.484287+0000","last_unstale":"2024-04-26T02:25:56.540378+0000","last_undegraded":"2024-04-26T02:25:56.540378+0000","last_fullsized":"2024-04-26T02:25:56.540378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'6605","last_scrub_stamp":"2024-04-26T02:25:56.540292+0000","last_deep_scrub":"24'6605","last_deep_scrub_stamp":"2024-04-26T02:25:56.540292+0000","last_clean_scrub_stamp":"2024-04-26T02:25:56.540292+0000","objects_scrubbed":28,"log_size":6605,"log_dups_size":0,"ondisk_log_size":6605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:09:35.471527+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6549,"num_write_kb":206212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"24'5751","reported_seq":5798,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:26:01.938890+0000","last_change":"2024-04-26T02:26:01.938954+0000","last_active":"2024-04-26T02:26:01.938890+0000","last_peered":"2024-04-26T02:26:01.938890+0000","last_clean":"2024-04-26T02:26:01.938890+0000","last_became_active":"2024-04-26T02:21:37.497700+0000","last_became_peered":"2024-04-26T02:21:37.497700+0000","last_unstale":"2024-04-26T02:26:01.938890+0000","last_undegraded":"2024-04-26T02:26:01.938890+0000","last_fullsized":"2024-04-26T02:26:01.938890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'5751","last_scrub_stamp":"2024-04-26T02:26:01.938728+0000","last_deep_scrub":"24'5751","last_deep_scrub_stamp":"2024-04-26T02:26:01.938728+0000","last_clean_scrub_stamp":"2024-04-26T02:26:01.938728+0000","objects_scrubbed":26,"log_size":5751,"log_dups_size":0,"ondisk_log_size":5751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:53:04.700290+0000","scrub_duration":487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":76,"num_read_kb":124,"num_write":5736,"num_write_kb":155565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"24'4088","reported_seq":4118,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:26:00.843011+0000","last_change":"2024-04-26T02:26:00.843086+0000","last_active":"2024-04-26T02:26:00.843011+0000","last_peered":"2024-04-26T02:26:00.843011+0000","last_clean":"2024-04-26T02:26:00.843011+0000","last_became_active":"2024-04-26T02:21:37.496863+0000","last_became_peered":"2024-04-26T02:21:37.496863+0000","last_unstale":"2024-04-26T02:26:00.843011+0000","last_undegraded":"2024-04-26T02:26:00.843011+0000","last_fullsized":"2024-04-26T02:26:00.843011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4088","last_scrub_stamp":"2024-04-26T02:26:00.842819+0000","last_deep_scrub":"24'4088","last_deep_scrub_stamp":"2024-04-26T02:26:00.842819+0000","last_clean_scrub_stamp":"2024-04-26T02:26:00.842819+0000","objects_scrubbed":29,"log_size":4088,"log_dups_size":0,"ondisk_log_size":4088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:41:19.843930+0000","scrub_duration":419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":4059,"num_write_kb":185415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"24'3980","reported_seq":4016,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:57.476949+0000","last_change":"2024-04-26T02:25:57.477017+0000","last_active":"2024-04-26T02:25:57.476949+0000","last_peered":"2024-04-26T02:25:57.476949+0000","last_clean":"2024-04-26T02:25:57.476949+0000","last_became_active":"2024-04-26T02:21:37.490474+0000","last_became_peered":"2024-04-26T02:21:37.490474+0000","last_unstale":"2024-04-26T02:25:57.476949+0000","last_undegraded":"2024-04-26T02:25:57.476949+0000","last_fullsized":"2024-04-26T02:25:57.476949+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'3980","last_scrub_stamp":"2024-04-26T02:25:57.476865+0000","last_deep_scrub":"24'3980","last_deep_scrub_stamp":"2024-04-26T02:25:57.476865+0000","last_clean_scrub_stamp":"2024-04-26T02:25:57.476865+0000","objects_scrubbed":30,"log_size":3980,"log_dups_size":0,"ondisk_log_size":3980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:12:47.096245+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120324096,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":34,"num_read_kb":151,"num_write":3926,"num_write_kb":190983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"24'6108","reported_seq":6141,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:57.275649+0000","last_change":"2024-04-26T02:25:57.275705+0000","last_active":"2024-04-26T02:25:57.275649+0000","last_peered":"2024-04-26T02:25:57.275649+0000","last_clean":"2024-04-26T02:25:57.275649+0000","last_became_active":"2024-04-26T02:21:37.485110+0000","last_became_peered":"2024-04-26T02:21:37.485110+0000","last_unstale":"2024-04-26T02:25:57.275649+0000","last_undegraded":"2024-04-26T02:25:57.275649+0000","last_fullsized":"2024-04-26T02:25:57.275649+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'6108","last_scrub_stamp":"2024-04-26T02:25:57.275523+0000","last_deep_scrub":"24'6108","last_deep_scrub_stamp":"2024-04-26T02:25:57.275523+0000","last_clean_scrub_stamp":"2024-04-26T02:25:57.275523+0000","objects_scrubbed":26,"log_size":6108,"log_dups_size":0,"ondisk_log_size":6108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:14:10.357476+0000","scrub_duration":431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":6064,"num_write_kb":187005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"24'5070","reported_seq":5116,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:59.352727+0000","last_change":"2024-04-26T02:25:59.352778+0000","last_active":"2024-04-26T02:25:59.352727+0000","last_peered":"2024-04-26T02:25:59.352727+0000","last_clean":"2024-04-26T02:25:59.352727+0000","last_became_active":"2024-04-26T02:21:37.488519+0000","last_became_peered":"2024-04-26T02:21:37.488519+0000","last_unstale":"2024-04-26T02:25:59.352727+0000","last_undegraded":"2024-04-26T02:25:59.352727+0000","last_fullsized":"2024-04-26T02:25:59.352727+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'5070","last_scrub_stamp":"2024-04-26T02:25:59.352648+0000","last_deep_scrub":"24'5070","last_deep_scrub_stamp":"2024-04-26T02:25:59.352648+0000","last_clean_scrub_stamp":"2024-04-26T02:25:59.352648+0000","objects_scrubbed":29,"log_size":5070,"log_dups_size":0,"ondisk_log_size":5070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:16:51.412894+0000","scrub_duration":443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117441024,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":5034,"num_write_kb":181716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"24'7186","reported_seq":7235,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:26:01.730219+0000","last_change":"2024-04-26T02:26:01.730297+0000","last_active":"2024-04-26T02:26:01.730219+0000","last_peered":"2024-04-26T02:26:01.730219+0000","last_clean":"2024-04-26T02:26:01.730219+0000","last_became_active":"2024-04-26T02:21:37.495694+0000","last_became_peered":"2024-04-26T02:21:37.495694+0000","last_unstale":"2024-04-26T02:26:01.730219+0000","last_undegraded":"2024-04-26T02:26:01.730219+0000","last_fullsized":"2024-04-26T02:26:01.730219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'7186","last_scrub_stamp":"2024-04-26T02:26:01.730126+0000","last_deep_scrub":"24'7186","last_deep_scrub_stamp":"2024-04-26T02:26:01.730126+0000","last_clean_scrub_stamp":"2024-04-26T02:26:01.730126+0000","objects_scrubbed":33,"log_size":7186,"log_dups_size":0,"ondisk_log_size":7186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:56:41.014153+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023936,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":80,"num_read_kb":65,"num_write":7184,"num_write_kb":197893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"24'8313","reported_seq":8348,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:56.599096+0000","last_change":"2024-04-26T02:25:56.599155+0000","last_active":"2024-04-26T02:25:56.599096+0000","last_peered":"2024-04-26T02:25:56.599096+0000","last_clean":"2024-04-26T02:25:56.599096+0000","last_became_active":"2024-04-26T02:21:37.490119+0000","last_became_peered":"2024-04-26T02:21:37.490119+0000","last_unstale":"2024-04-26T02:25:56.599096+0000","last_undegraded":"2024-04-26T02:25:56.599096+0000","last_fullsized":"2024-04-26T02:25:56.599096+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'8313","last_scrub_stamp":"2024-04-26T02:25:56.599000+0000","last_deep_scrub":"24'8313","last_deep_scrub_stamp":"2024-04-26T02:25:56.599000+0000","last_clean_scrub_stamp":"2024-04-26T02:25:56.599000+0000","objects_scrubbed":33,"log_size":8313,"log_dups_size":0,"ondisk_log_size":8313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:03:41.468031+0000","scrub_duration":575,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137060352,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":8291,"num_write_kb":229215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"24'3776","reported_seq":3802,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:26:02.707367+0000","last_change":"2024-04-26T02:26:02.707507+0000","last_active":"2024-04-26T02:26:02.707367+0000","last_peered":"2024-04-26T02:26:02.707367+0000","last_clean":"2024-04-26T02:26:02.707367+0000","last_became_active":"2024-04-26T02:21:37.506309+0000","last_became_peered":"2024-04-26T02:21:37.506309+0000","last_unstale":"2024-04-26T02:26:02.707367+0000","last_undegraded":"2024-04-26T02:26:02.707367+0000","last_fullsized":"2024-04-26T02:26:02.707367+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'3776","last_scrub_stamp":"2024-04-26T02:26:02.707154+0000","last_deep_scrub":"24'3776","last_deep_scrub_stamp":"2024-04-26T02:26:02.707154+0000","last_clean_scrub_stamp":"2024-04-26T02:26:02.707154+0000","objects_scrubbed":19,"log_size":3776,"log_dups_size":0,"ondisk_log_size":3776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:13:24.102894+0000","scrub_duration":275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3724,"num_write_kb":119684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"24'7207","reported_seq":7234,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:26:01.486334+0000","last_change":"2024-04-26T02:26:01.486429+0000","last_active":"2024-04-26T02:26:01.486334+0000","last_peered":"2024-04-26T02:26:01.486334+0000","last_clean":"2024-04-26T02:26:01.486334+0000","last_became_active":"2024-04-26T02:21:37.486868+0000","last_became_peered":"2024-04-26T02:21:37.486868+0000","last_unstale":"2024-04-26T02:26:01.486334+0000","last_undegraded":"2024-04-26T02:26:01.486334+0000","last_fullsized":"2024-04-26T02:26:01.486334+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'7207","last_scrub_stamp":"2024-04-26T02:26:01.486156+0000","last_deep_scrub":"24'7207","last_deep_scrub_stamp":"2024-04-26T02:26:01.486156+0000","last_clean_scrub_stamp":"2024-04-26T02:26:01.486156+0000","objects_scrubbed":30,"log_size":7207,"log_dups_size":0,"ondisk_log_size":7207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:04:35.348650+0000","scrub_duration":531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7176,"num_write_kb":198373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"24'4558","reported_seq":4584,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:58.222515+0000","last_change":"2024-04-26T02:25:58.222623+0000","last_active":"2024-04-26T02:25:58.222515+0000","last_peered":"2024-04-26T02:25:58.222515+0000","last_clean":"2024-04-26T02:25:58.222515+0000","last_became_active":"2024-04-26T02:21:37.487980+0000","last_became_peered":"2024-04-26T02:21:37.487980+0000","last_unstale":"2024-04-26T02:25:58.222515+0000","last_undegraded":"2024-04-26T02:25:58.222515+0000","last_fullsized":"2024-04-26T02:25:58.222515+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4558","last_scrub_stamp":"2024-04-26T02:25:58.222416+0000","last_deep_scrub":"24'4558","last_deep_scrub_stamp":"2024-04-26T02:25:58.222416+0000","last_clean_scrub_stamp":"2024-04-26T02:25:58.222416+0000","objects_scrubbed":22,"log_size":4558,"log_dups_size":0,"ondisk_log_size":4558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:58:07.155322+0000","scrub_duration":379,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4488,"num_write_kb":107395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:57.342754+0000","last_change":"2024-04-26T02:25:57.342845+0000","last_active":"2024-04-26T02:25:57.342754+0000","last_peered":"2024-04-26T02:25:57.342754+0000","last_clean":"2024-04-26T02:25:57.342754+0000","last_became_active":"2024-04-26T02:21:31.428657+0000","last_became_peered":"2024-04-26T02:21:31.428657+0000","last_unstale":"2024-04-26T02:25:57.342754+0000","last_undegraded":"2024-04-26T02:25:57.342754+0000","last_fullsized":"2024-04-26T02:25:57.342754+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:25:57.342458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:25:57.342458+0000","last_clean_scrub_stamp":"2024-04-26T02:25:57.342458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:39:54.846217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"24'4911","reported_seq":4946,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:54.222676+0000","last_change":"2024-04-26T02:25:54.222676+0000","last_active":"2024-04-26T02:25:54.222676+0000","last_peered":"2024-04-26T02:25:54.222676+0000","last_clean":"2024-04-26T02:25:54.222676+0000","last_became_active":"2024-04-26T02:21:37.481620+0000","last_became_peered":"2024-04-26T02:21:37.481620+0000","last_unstale":"2024-04-26T02:25:54.222676+0000","last_undegraded":"2024-04-26T02:25:54.222676+0000","last_fullsized":"2024-04-26T02:25:54.222676+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4911","last_scrub_stamp":"2024-04-26T02:25:54.222501+0000","last_deep_scrub":"24'4911","last_deep_scrub_stamp":"2024-04-26T02:25:54.222501+0000","last_clean_scrub_stamp":"2024-04-26T02:25:54.222501+0000","objects_scrubbed":24,"log_size":4911,"log_dups_size":0,"ondisk_log_size":4911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:18.375457+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96600064,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":4875,"num_write_kb":131621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:55.872605+0000","last_change":"2024-04-26T02:25:55.872640+0000","last_active":"2024-04-26T02:25:55.872605+0000","last_peered":"2024-04-26T02:25:55.872605+0000","last_clean":"2024-04-26T02:25:55.872605+0000","last_became_active":"2024-04-26T02:21:31.431958+0000","last_became_peered":"2024-04-26T02:21:31.431958+0000","last_unstale":"2024-04-26T02:25:55.872605+0000","last_undegraded":"2024-04-26T02:25:55.872605+0000","last_fullsized":"2024-04-26T02:25:55.872605+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:25:55.872467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:25:55.872467+0000","last_clean_scrub_stamp":"2024-04-26T02:25:55.872467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:42:50.043309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"24'4252","reported_seq":4280,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:59.564300+0000","last_change":"2024-04-26T02:25:59.564300+0000","last_active":"2024-04-26T02:25:59.564300+0000","last_peered":"2024-04-26T02:25:59.564300+0000","last_clean":"2024-04-26T02:25:59.564300+0000","last_became_active":"2024-04-26T02:21:37.487980+0000","last_became_peered":"2024-04-26T02:21:37.487980+0000","last_unstale":"2024-04-26T02:25:59.564300+0000","last_undegraded":"2024-04-26T02:25:59.564300+0000","last_fullsized":"2024-04-26T02:25:59.564300+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4252","last_scrub_stamp":"2024-04-26T02:25:59.564123+0000","last_deep_scrub":"24'4252","last_deep_scrub_stamp":"2024-04-26T02:25:59.564123+0000","last_clean_scrub_stamp":"2024-04-26T02:25:59.564123+0000","objects_scrubbed":28,"log_size":4252,"log_dups_size":0,"ondisk_log_size":4252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:37:43.012708+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4175,"num_write_kb":201729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:58.920657+0000","last_change":"2024-04-26T02:25:58.920680+0000","last_active":"2024-04-26T02:25:58.920657+0000","last_peered":"2024-04-26T02:25:58.920657+0000","last_clean":"2024-04-26T02:25:58.920657+0000","last_became_active":"2024-04-26T02:21:31.430227+0000","last_became_peered":"2024-04-26T02:21:31.430227+0000","last_unstale":"2024-04-26T02:25:58.920657+0000","last_undegraded":"2024-04-26T02:25:58.920657+0000","last_fullsized":"2024-04-26T02:25:58.920657+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:25:58.920560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:25:58.920560+0000","last_clean_scrub_stamp":"2024-04-26T02:25:58.920560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:54:43.551290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"24'6146","reported_seq":6179,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:57.185273+0000","last_change":"2024-04-26T02:25:57.185301+0000","last_active":"2024-04-26T02:25:57.185273+0000","last_peered":"2024-04-26T02:25:57.185273+0000","last_clean":"2024-04-26T02:25:57.185273+0000","last_became_active":"2024-04-26T02:21:37.482894+0000","last_became_peered":"2024-04-26T02:21:37.482894+0000","last_unstale":"2024-04-26T02:25:57.185273+0000","last_undegraded":"2024-04-26T02:25:57.185273+0000","last_fullsized":"2024-04-26T02:25:57.185273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'6146","last_scrub_stamp":"2024-04-26T02:25:57.185204+0000","last_deep_scrub":"24'6146","last_deep_scrub_stamp":"2024-04-26T02:25:57.185204+0000","last_clean_scrub_stamp":"2024-04-26T02:25:57.185204+0000","objects_scrubbed":25,"log_size":6146,"log_dups_size":0,"ondisk_log_size":6146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:08:07.711355+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99354976,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":6134,"num_write_kb":151141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:54.876012+0000","last_change":"2024-04-26T02:25:54.876069+0000","last_active":"2024-04-26T02:25:54.876012+0000","last_peered":"2024-04-26T02:25:54.876012+0000","last_clean":"2024-04-26T02:25:54.876012+0000","last_became_active":"2024-04-26T02:21:31.431071+0000","last_became_peered":"2024-04-26T02:21:31.431071+0000","last_unstale":"2024-04-26T02:25:54.876012+0000","last_undegraded":"2024-04-26T02:25:54.876012+0000","last_fullsized":"2024-04-26T02:25:54.876012+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:25:54.875820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:25:54.875820+0000","last_clean_scrub_stamp":"2024-04-26T02:25:54.875820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:17:31.825034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"24'4211","reported_seq":4268,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:58.859684+0000","last_change":"2024-04-26T02:25:58.859760+0000","last_active":"2024-04-26T02:25:58.859684+0000","last_peered":"2024-04-26T02:25:58.859684+0000","last_clean":"2024-04-26T02:25:58.859684+0000","last_became_active":"2024-04-26T02:21:37.490105+0000","last_became_peered":"2024-04-26T02:21:37.490105+0000","last_unstale":"2024-04-26T02:25:58.859684+0000","last_undegraded":"2024-04-26T02:25:58.859684+0000","last_fullsized":"2024-04-26T02:25:58.859684+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4211","last_scrub_stamp":"2024-04-26T02:25:58.859540+0000","last_deep_scrub":"24'4211","last_deep_scrub_stamp":"2024-04-26T02:25:58.859540+0000","last_clean_scrub_stamp":"2024-04-26T02:25:58.859540+0000","objects_scrubbed":31,"log_size":4211,"log_dups_size":0,"ondisk_log_size":4211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:27:31.796484+0000","scrub_duration":479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121896960,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":122,"num_read_kb":98,"num_write":4227,"num_write_kb":167240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"24'2980","reported_seq":3009,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:59.157442+0000","last_change":"2024-04-26T02:25:59.157630+0000","last_active":"2024-04-26T02:25:59.157442+0000","last_peered":"2024-04-26T02:25:59.157442+0000","last_clean":"2024-04-26T02:25:59.157442+0000","last_became_active":"2024-04-26T02:21:37.491249+0000","last_became_peered":"2024-04-26T02:21:37.491249+0000","last_unstale":"2024-04-26T02:25:59.157442+0000","last_undegraded":"2024-04-26T02:25:59.157442+0000","last_fullsized":"2024-04-26T02:25:59.157442+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'2980","last_scrub_stamp":"2024-04-26T02:25:59.157223+0000","last_deep_scrub":"24'2980","last_deep_scrub_stamp":"2024-04-26T02:25:59.157223+0000","last_clean_scrub_stamp":"2024-04-26T02:25:59.157223+0000","objects_scrubbed":18,"log_size":2980,"log_dups_size":0,"ondisk_log_size":2980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:23:55.221272+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74842112,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":7,"num_write":2966,"num_write_kb":163586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:56.375086+0000","last_change":"2024-04-26T02:25:56.375106+0000","last_active":"2024-04-26T02:25:56.375086+0000","last_peered":"2024-04-26T02:25:56.375086+0000","last_clean":"2024-04-26T02:25:56.375086+0000","last_became_active":"2024-04-26T02:21:32.425189+0000","last_became_peered":"2024-04-26T02:21:32.425189+0000","last_unstale":"2024-04-26T02:25:56.375086+0000","last_undegraded":"2024-04-26T02:25:56.375086+0000","last_fullsized":"2024-04-26T02:25:56.375086+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:25:56.375013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:25:56.375013+0000","last_clean_scrub_stamp":"2024-04-26T02:25:56.375013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:04:28.083893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"24'3505","reported_seq":3538,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:58.230318+0000","last_change":"2024-04-26T02:25:58.230454+0000","last_active":"2024-04-26T02:25:58.230318+0000","last_peered":"2024-04-26T02:25:58.230318+0000","last_clean":"2024-04-26T02:25:58.230318+0000","last_became_active":"2024-04-26T02:21:37.485432+0000","last_became_peered":"2024-04-26T02:21:37.485432+0000","last_unstale":"2024-04-26T02:25:58.230318+0000","last_undegraded":"2024-04-26T02:25:58.230318+0000","last_fullsized":"2024-04-26T02:25:58.230318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'3505","last_scrub_stamp":"2024-04-26T02:25:58.230210+0000","last_deep_scrub":"24'3505","last_deep_scrub_stamp":"2024-04-26T02:25:58.230210+0000","last_clean_scrub_stamp":"2024-04-26T02:25:58.230210+0000","objects_scrubbed":22,"log_size":3505,"log_dups_size":0,"ondisk_log_size":3505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:06:48.765656+0000","scrub_duration":351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86706520,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":3452,"num_write_kb":123177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:55.614074+0000","last_change":"2024-04-26T02:25:55.614094+0000","last_active":"2024-04-26T02:25:55.614074+0000","last_peered":"2024-04-26T02:25:55.614074+0000","last_clean":"2024-04-26T02:25:55.614074+0000","last_became_active":"2024-04-26T02:21:31.436226+0000","last_became_peered":"2024-04-26T02:21:31.436226+0000","last_unstale":"2024-04-26T02:25:55.614074+0000","last_undegraded":"2024-04-26T02:25:55.614074+0000","last_fullsized":"2024-04-26T02:25:55.614074+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:25:55.614000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:25:55.614000+0000","last_clean_scrub_stamp":"2024-04-26T02:25:55.614000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:36:35.612494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"24'4484","reported_seq":4513,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:59.544863+0000","last_change":"2024-04-26T02:25:59.544905+0000","last_active":"2024-04-26T02:25:59.544863+0000","last_peered":"2024-04-26T02:25:59.544863+0000","last_clean":"2024-04-26T02:25:59.544863+0000","last_became_active":"2024-04-26T02:21:37.493922+0000","last_became_peered":"2024-04-26T02:21:37.493922+0000","last_unstale":"2024-04-26T02:25:59.544863+0000","last_undegraded":"2024-04-26T02:25:59.544863+0000","last_fullsized":"2024-04-26T02:25:59.544863+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4484","last_scrub_stamp":"2024-04-26T02:25:59.544783+0000","last_deep_scrub":"24'4484","last_deep_scrub_stamp":"2024-04-26T02:25:59.544783+0000","last_clean_scrub_stamp":"2024-04-26T02:25:59.544783+0000","objects_scrubbed":28,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:44:16.773800+0000","scrub_duration":443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113312264,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":4436,"num_write_kb":167962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:59.960605+0000","last_change":"2024-04-26T02:25:59.960715+0000","last_active":"2024-04-26T02:25:59.960605+0000","last_peered":"2024-04-26T02:25:59.960605+0000","last_clean":"2024-04-26T02:25:59.960605+0000","last_became_active":"2024-04-26T02:21:31.432395+0000","last_became_peered":"2024-04-26T02:21:31.432395+0000","last_unstale":"2024-04-26T02:25:59.960605+0000","last_undegraded":"2024-04-26T02:25:59.960605+0000","last_fullsized":"2024-04-26T02:25:59.960605+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:25:59.960390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:25:59.960390+0000","last_clean_scrub_stamp":"2024-04-26T02:25:59.960390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:11:32.619260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"24'4953","reported_seq":4993,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:57.512496+0000","last_change":"2024-04-26T02:25:57.512595+0000","last_active":"2024-04-26T02:25:57.512496+0000","last_peered":"2024-04-26T02:25:57.512496+0000","last_clean":"2024-04-26T02:25:57.512496+0000","last_became_active":"2024-04-26T02:21:37.484172+0000","last_became_peered":"2024-04-26T02:21:37.484172+0000","last_unstale":"2024-04-26T02:25:57.512496+0000","last_undegraded":"2024-04-26T02:25:57.512496+0000","last_fullsized":"2024-04-26T02:25:57.512496+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4953","last_scrub_stamp":"2024-04-26T02:25:57.512409+0000","last_deep_scrub":"24'4953","last_deep_scrub_stamp":"2024-04-26T02:25:57.512409+0000","last_clean_scrub_stamp":"2024-04-26T02:25:57.512409+0000","objects_scrubbed":23,"log_size":4953,"log_dups_size":0,"ondisk_log_size":4953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:28:40.028461+0000","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91030032,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":49,"num_read_kb":40,"num_write":4960,"num_write_kb":192278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:57.410391+0000","last_change":"2024-04-26T02:25:57.410473+0000","last_active":"2024-04-26T02:25:57.410391+0000","last_peered":"2024-04-26T02:25:57.410391+0000","last_clean":"2024-04-26T02:25:57.410391+0000","last_became_active":"2024-04-26T02:21:33.340937+0000","last_became_peered":"2024-04-26T02:21:33.340937+0000","last_unstale":"2024-04-26T02:25:57.410391+0000","last_undegraded":"2024-04-26T02:25:57.410391+0000","last_fullsized":"2024-04-26T02:25:57.410391+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T02:25:57.410200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T02:25:57.410200+0000","last_clean_scrub_stamp":"2024-04-26T02:25:57.410200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:53.286910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"24'119","reported_seq":191,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:57.940060+0000","last_change":"2024-04-26T02:25:57.940195+0000","last_active":"2024-04-26T02:25:57.940060+0000","last_peered":"2024-04-26T02:25:57.940060+0000","last_clean":"2024-04-26T02:25:57.940060+0000","last_became_active":"2024-04-26T02:21:29.432363+0000","last_became_peered":"2024-04-26T02:21:29.432363+0000","last_unstale":"2024-04-26T02:25:57.940060+0000","last_undegraded":"2024-04-26T02:25:57.940060+0000","last_fullsized":"2024-04-26T02:25:57.940060+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'119","last_scrub_stamp":"2024-04-26T02:25:57.939861+0000","last_deep_scrub":"24'119","last_deep_scrub_stamp":"2024-04-26T02:25:57.939861+0000","last_clean_scrub_stamp":"2024-04-26T02:25:57.939861+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:12:54.999789+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"24'4929","reported_seq":4957,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:58.660394+0000","last_change":"2024-04-26T02:25:58.660394+0000","last_active":"2024-04-26T02:25:58.660394+0000","last_peered":"2024-04-26T02:25:58.660394+0000","last_clean":"2024-04-26T02:25:58.660394+0000","last_became_active":"2024-04-26T02:21:37.486673+0000","last_became_peered":"2024-04-26T02:21:37.486673+0000","last_unstale":"2024-04-26T02:25:58.660394+0000","last_undegraded":"2024-04-26T02:25:58.660394+0000","last_fullsized":"2024-04-26T02:25:58.660394+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4929","last_scrub_stamp":"2024-04-26T02:25:58.660234+0000","last_deep_scrub":"24'4929","last_deep_scrub_stamp":"2024-04-26T02:25:58.660234+0000","last_clean_scrub_stamp":"2024-04-26T02:25:58.660234+0000","objects_scrubbed":35,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:29:27.861601+0000","scrub_duration":519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4819,"num_write_kb":226567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"24'9688","reported_seq":9826,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:56.371208+0000","last_change":"2024-04-26T02:25:56.371208+0000","last_active":"2024-04-26T02:25:56.371208+0000","last_peered":"2024-04-26T02:25:56.371208+0000","last_clean":"2024-04-26T02:25:56.371208+0000","last_became_active":"2024-04-26T02:21:37.476072+0000","last_became_peered":"2024-04-26T02:21:37.476072+0000","last_unstale":"2024-04-26T02:25:56.371208+0000","last_undegraded":"2024-04-26T02:25:56.371208+0000","last_fullsized":"2024-04-26T02:25:56.371208+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'9688","last_scrub_stamp":"2024-04-26T02:25:56.371074+0000","last_deep_scrub":"24'9688","last_deep_scrub_stamp":"2024-04-26T02:25:56.371074+0000","last_clean_scrub_stamp":"2024-04-26T02:25:56.371074+0000","objects_scrubbed":34,"log_size":9688,"log_dups_size":0,"ondisk_log_size":9688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:29:44.715224+0000","scrub_duration":575,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138481144,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":373,"num_read_kb":339,"num_write":7867,"num_write_kb":214107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"24'5803","reported_seq":5940,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:58.870874+0000","last_change":"2024-04-26T02:25:58.870937+0000","last_active":"2024-04-26T02:25:58.870874+0000","last_peered":"2024-04-26T02:25:58.870874+0000","last_clean":"2024-04-26T02:25:58.870874+0000","last_became_active":"2024-04-26T02:21:37.488724+0000","last_became_peered":"2024-04-26T02:21:37.488724+0000","last_unstale":"2024-04-26T02:25:58.870874+0000","last_undegraded":"2024-04-26T02:25:58.870874+0000","last_fullsized":"2024-04-26T02:25:58.870874+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'5803","last_scrub_stamp":"2024-04-26T02:25:58.870796+0000","last_deep_scrub":"24'5803","last_deep_scrub_stamp":"2024-04-26T02:25:58.870796+0000","last_clean_scrub_stamp":"2024-04-26T02:25:58.870796+0000","objects_scrubbed":34,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:17:18.159680+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134480400,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":132,"num_read_kb":124,"num_write":5774,"num_write_kb":201317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"24'3925","reported_seq":3952,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:26:00.537639+0000","last_change":"2024-04-26T02:26:00.537639+0000","last_active":"2024-04-26T02:26:00.537639+0000","last_peered":"2024-04-26T02:26:00.537639+0000","last_clean":"2024-04-26T02:26:00.537639+0000","last_became_active":"2024-04-26T02:21:37.491785+0000","last_became_peered":"2024-04-26T02:21:37.491785+0000","last_unstale":"2024-04-26T02:26:00.537639+0000","last_undegraded":"2024-04-26T02:26:00.537639+0000","last_fullsized":"2024-04-26T02:26:00.537639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'3925","last_scrub_stamp":"2024-04-26T02:26:00.537230+0000","last_deep_scrub":"24'3925","last_deep_scrub_stamp":"2024-04-26T02:26:00.537230+0000","last_clean_scrub_stamp":"2024-04-26T02:26:00.537230+0000","objects_scrubbed":22,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:58.739264+0000","scrub_duration":347,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88842240,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3901,"num_write_kb":140906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"24'3945","reported_seq":3975,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:59.728059+0000","last_change":"2024-04-26T02:25:59.728125+0000","last_active":"2024-04-26T02:25:59.728059+0000","last_peered":"2024-04-26T02:25:59.728059+0000","last_clean":"2024-04-26T02:25:59.728059+0000","last_became_active":"2024-04-26T02:21:37.490836+0000","last_became_peered":"2024-04-26T02:21:37.490836+0000","last_unstale":"2024-04-26T02:25:59.728059+0000","last_undegraded":"2024-04-26T02:25:59.728059+0000","last_fullsized":"2024-04-26T02:25:59.728059+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'3945","last_scrub_stamp":"2024-04-26T02:25:59.727937+0000","last_deep_scrub":"24'3945","last_deep_scrub_stamp":"2024-04-26T02:25:59.727937+0000","last_clean_scrub_stamp":"2024-04-26T02:25:59.727937+0000","objects_scrubbed":25,"log_size":3945,"log_dups_size":0,"ondisk_log_size":3945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:35:18.760770+0000","scrub_duration":351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100672016,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3880,"num_write_kb":189331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"24'3171","reported_seq":3506,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:26:06.231166+0000","last_change":"2024-04-26T02:26:06.231166+0000","last_active":"2024-04-26T02:26:06.231166+0000","last_peered":"2024-04-26T02:26:06.231166+0000","last_clean":"2024-04-26T02:26:06.231166+0000","last_became_active":"2024-04-26T02:21:37.500919+0000","last_became_peered":"2024-04-26T02:21:37.500919+0000","last_unstale":"2024-04-26T02:26:06.231166+0000","last_undegraded":"2024-04-26T02:26:06.231166+0000","last_fullsized":"2024-04-26T02:26:06.231166+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'3171","last_scrub_stamp":"2024-04-26T02:26:06.230799+0000","last_deep_scrub":"24'3171","last_deep_scrub_stamp":"2024-04-26T02:26:06.230799+0000","last_clean_scrub_stamp":"2024-04-26T02:26:06.230799+0000","objects_scrubbed":22,"log_size":3171,"log_dups_size":0,"ondisk_log_size":3171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:11:45.094423+0000","scrub_duration":299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84148752,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1028,"num_read_kb":921,"num_write":3342,"num_write_kb":128651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"24'4894","reported_seq":5020,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:26:04.381696+0000","last_change":"2024-04-26T02:26:04.381696+0000","last_active":"2024-04-26T02:26:04.381696+0000","last_peered":"2024-04-26T02:26:04.381696+0000","last_clean":"2024-04-26T02:26:04.381696+0000","last_became_active":"2024-04-26T02:21:37.489359+0000","last_became_peered":"2024-04-26T02:21:37.489359+0000","last_unstale":"2024-04-26T02:26:04.381696+0000","last_undegraded":"2024-04-26T02:26:04.381696+0000","last_fullsized":"2024-04-26T02:26:04.381696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4894","last_scrub_stamp":"2024-04-26T02:26:04.381423+0000","last_deep_scrub":"24'4894","last_deep_scrub_stamp":"2024-04-26T02:26:04.381423+0000","last_clean_scrub_stamp":"2024-04-26T02:26:04.381423+0000","objects_scrubbed":32,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:37.001090+0000","scrub_duration":511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130154496,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":323,"num_read_kb":315,"num_write":5125,"num_write_kb":178670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"24'3737","reported_seq":3764,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:26:03.202985+0000","last_change":"2024-04-26T02:26:03.202985+0000","last_active":"2024-04-26T02:26:03.202985+0000","last_peered":"2024-04-26T02:26:03.202985+0000","last_clean":"2024-04-26T02:26:03.202985+0000","last_became_active":"2024-04-26T02:21:37.488144+0000","last_became_peered":"2024-04-26T02:21:37.488144+0000","last_unstale":"2024-04-26T02:26:03.202985+0000","last_undegraded":"2024-04-26T02:26:03.202985+0000","last_fullsized":"2024-04-26T02:26:03.202985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'3737","last_scrub_stamp":"2024-04-26T02:26:03.202621+0000","last_deep_scrub":"24'3737","last_deep_scrub_stamp":"2024-04-26T02:26:03.202621+0000","last_clean_scrub_stamp":"2024-04-26T02:26:03.202621+0000","objects_scrubbed":22,"log_size":3737,"log_dups_size":0,"ondisk_log_size":3737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:41:47.558659+0000","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":168964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"24'3613","reported_seq":3643,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:59.784343+0000","last_change":"2024-04-26T02:25:59.784434+0000","last_active":"2024-04-26T02:25:59.784343+0000","last_peered":"2024-04-26T02:25:59.784343+0000","last_clean":"2024-04-26T02:25:59.784343+0000","last_became_active":"2024-04-26T02:21:37.494442+0000","last_became_peered":"2024-04-26T02:21:37.494442+0000","last_unstale":"2024-04-26T02:25:59.784343+0000","last_undegraded":"2024-04-26T02:25:59.784343+0000","last_fullsized":"2024-04-26T02:25:59.784343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'3613","last_scrub_stamp":"2024-04-26T02:25:59.784143+0000","last_deep_scrub":"24'3613","last_deep_scrub_stamp":"2024-04-26T02:25:59.784143+0000","last_clean_scrub_stamp":"2024-04-26T02:25:59.784143+0000","objects_scrubbed":30,"log_size":3613,"log_dups_size":0,"ondisk_log_size":3613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:31:29.278302+0000","scrub_duration":435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":3543,"num_write_kb":168836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"24'4343","reported_seq":4370,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:26:05.410587+0000","last_change":"2024-04-26T02:26:05.410682+0000","last_active":"2024-04-26T02:26:05.410587+0000","last_peered":"2024-04-26T02:26:05.410587+0000","last_clean":"2024-04-26T02:26:05.410587+0000","last_became_active":"2024-04-26T02:21:37.497599+0000","last_became_peered":"2024-04-26T02:21:37.497599+0000","last_unstale":"2024-04-26T02:26:05.410587+0000","last_undegraded":"2024-04-26T02:26:05.410587+0000","last_fullsized":"2024-04-26T02:26:05.410587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4343","last_scrub_stamp":"2024-04-26T02:26:05.410402+0000","last_deep_scrub":"24'4343","last_deep_scrub_stamp":"2024-04-26T02:26:05.410402+0000","last_clean_scrub_stamp":"2024-04-26T02:26:05.410402+0000","objects_scrubbed":33,"log_size":4343,"log_dups_size":0,"ondisk_log_size":4343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:30:41.838544+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4304,"num_write_kb":209927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"24'4362","reported_seq":4395,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:25:55.416724+0000","last_change":"2024-04-26T02:25:55.416805+0000","last_active":"2024-04-26T02:25:55.416724+0000","last_peered":"2024-04-26T02:25:55.416724+0000","last_clean":"2024-04-26T02:25:55.416724+0000","last_became_active":"2024-04-26T02:21:37.486967+0000","last_became_peered":"2024-04-26T02:21:37.486967+0000","last_unstale":"2024-04-26T02:25:55.416724+0000","last_undegraded":"2024-04-26T02:25:55.416724+0000","last_fullsized":"2024-04-26T02:25:55.416724+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4362","last_scrub_stamp":"2024-04-26T02:25:55.416588+0000","last_deep_scrub":"24'4362","last_deep_scrub_stamp":"2024-04-26T02:25:55.416588+0000","last_clean_scrub_stamp":"2024-04-26T02:25:55.416588+0000","objects_scrubbed":32,"log_size":4362,"log_dups_size":0,"ondisk_log_size":4362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:40:11.880327+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127401984,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23,"num_read_kb":80,"num_write":4288,"num_write_kb":199362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"24'8918","reported_seq":8973,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-26T02:26:02.520459+0000","last_change":"2024-04-26T02:26:02.520528+0000","last_active":"2024-04-26T02:26:02.520459+0000","last_peered":"2024-04-26T02:26:02.520459+0000","last_clean":"2024-04-26T02:26:02.520459+0000","last_became_active":"2024-04-26T02:21:37.488117+0000","last_became_peered":"2024-04-26T02:21:37.488117+0000","last_unstale":"2024-04-26T02:26:02.520459+0000","last_undegraded":"2024-04-26T02:26:02.520459+0000","last_fullsized":"2024-04-26T02:26:02.520459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'8918","last_scrub_stamp":"2024-04-26T02:26:02.520299+0000","last_deep_scrub":"24'8918","last_deep_scrub_stamp":"2024-04-26T02:26:02.520299+0000","last_clean_scrub_stamp":"2024-04-26T02:26:02.520299+0000","objects_scrubbed":33,"log_size":8918,"log_dups_size":0,"ondisk_log_size":8918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:29:56.702173+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128582144,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":108,"num_read_kb":91,"num_write":8909,"num_write_kb":228167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3568390392,"num_objects":877,"num_object_clones":0,"num_object_copies":1754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":877,"num_whiteouts":0,"num_read":2559,"num_read_kb":2575,"num_write":159728,"num_write_kb":5636581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6797877248,"data_stored":6835430520,"data_compressed":194293810,"data_compressed_allocated":202072064,"data_compressed_original":243892224,"omap_allocated":0,"internal_metadata":0},"log_size":162180,"ondisk_log_size":162180,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11166,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542204,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1412416,"kb_used_data":1038392,"kb_used_omap":1,"kb_used_meta":374014,"kb_avail":92959424,"statfs":{"total":96636764160,"available":95190450176,"internally_reserved":0,"allocated":1063313408,"data_stored":1066328537,"data_compressed":20319008,"data_compressed_allocated":21159936,"data_compressed_original":25022464,"omap_allocated":1590,"internal_metadata":382990794},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542205,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1148340,"kb_used_data":864556,"kb_used_omap":1,"kb_used_meta":283774,"kb_avail":93223500,"statfs":{"total":96636764160,"available":95460864000,"internally_reserved":0,"allocated":885305344,"data_stored":888268833,"data_compressed":21380520,"data_compressed_allocated":22167552,"data_compressed_original":25878528,"omap_allocated":1595,"internal_metadata":290585029},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542204,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1944992,"kb_used_data":1386328,"kb_used_omap":1,"kb_used_meta":558654,"kb_avail":92426848,"statfs":{"total":96636764160,"available":94645092352,"internally_reserved":0,"allocated":1419599872,"data_stored":1427699009,"data_compressed":38724772,"data_compressed_allocated":40361984,"data_compressed_original":49508352,"omap_allocated":1595,"internal_metadata":572062149},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542206,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1450516,"kb_used_data":1076492,"kb_used_omap":1,"kb_used_meta":374014,"kb_avail":92921324,"statfs":{"total":96636764160,"available":95151435776,"internally_reserved":0,"allocated":1102327808,"data_stored":1108250049,"data_compressed":31089397,"data_compressed_allocated":32296960,"data_compressed_original":39075840,"omap_allocated":1592,"internal_metadata":382990792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542204,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":885776,"kb_used_data":615880,"kb_used_omap":1,"kb_used_meta":269886,"kb_avail":93486064,"statfs":{"total":96636764160,"available":95729729536,"internally_reserved":0,"allocated":630661120,"data_stored":637301965,"data_compressed":27771469,"data_compressed_allocated":28930048,"data_compressed_original":36147200,"omap_allocated":1595,"internal_metadata":276363717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542204,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1370760,"kb_used_data":1018944,"kb_used_omap":1,"kb_used_meta":351806,"kb_avail":93001080,"statfs":{"total":96636764160,"available":95233105920,"internally_reserved":0,"allocated":1043398656,"data_stored":1049611477,"data_compressed":28674963,"data_compressed_allocated":29855744,"data_compressed_original":36777984,"omap_allocated":1595,"internal_metadata":360249797},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574909,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464500,"kb_used_data":343084,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93907340,"statfs":{"total":96636764160,"available":96161116160,"internally_reserved":0,"allocated":351318016,"data_stored":352441097,"data_compressed":9222947,"data_compressed_allocated":9637888,"data_compressed_original":11067392,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574909,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466668,"kb_used_data":296740,"kb_used_omap":1,"kb_used_meta":169918,"kb_avail":93905172,"statfs":{"total":96636764160,"available":96158896128,"internally_reserved":0,"allocated":303861760,"data_stored":307213657,"data_compressed":17121900,"data_compressed_allocated":17727488,"data_compressed_original":21438464,"omap_allocated":1595,"internal_metadata":173996485},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5583,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1043120128,"data_stored":1049004268,"data_compressed":28669380,"data_compressed_allocated":29822976,"data_compressed_original":36265984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":630435840,"data_stored":637223684,"data_compressed":27771469,"data_compressed_allocated":28930048,"data_compressed_original":36147200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303636480,"data_stored":307135376,"data_compressed":17121900,"data_compressed_allocated":17727488,"data_compressed_original":21438464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1102102528,"data_stored":1108171768,"data_compressed":31089397,"data_compressed_allocated":32296960,"data_compressed_original":39075840,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1419374592,"data_stored":1427620728,"data_compressed":38724772,"data_compressed_allocated":40361984,"data_compressed_original":49508352,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":885080064,"data_stored":888190552,"data_compressed":21380520,"data_compressed_allocated":22167552,"data_compressed_original":25878528,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":351092736,"data_stored":352362816,"data_compressed":9222947,"data_compressed_allocated":9637888,"data_compressed_original":11067392,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1063034880,"data_stored":1065721328,"data_compressed":20313425,"data_compressed_allocated":21127168,"data_compressed_original":24510464,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T02:26:17.978 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-26T02:26:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76fbbc8640 1 Processor -- start 2024-04-26T02:26:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76fbbc8640 1 -- start start 2024-04-26T02:26:18.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76fbbc8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76f40a9f00 con 0x7f76f4072d30 2024-04-26T02:26:18.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76fbbc8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76f40aa1e0 con 0x7f76f41360a0 2024-04-26T02:26:18.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76fbbc8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76f40708b0 con 0x7f76f4135cc0 2024-04-26T02:26:18.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76fa165640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f76f4072d30 0x7f76f414ad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:34718/0 (socket says 172.21.15.87:34718) 2024-04-26T02:26:18.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76fa165640 1 -- 172.21.15.87:0/379804153 learned_addr learned my addr 172.21.15.87:0/379804153 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:26:18.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76f9964640 1 --1- 172.21.15.87:0/379804153 >> v1:172.21.15.87:6790/0 conn(0x7f76f4135cc0 0x7f76f40701a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:54368/0 (socket says 172.21.15.87:54368) 2024-04-26T02:26:18.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76f8962640 1 -- 172.21.15.87:0/379804153 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 289872709 0 0) 0x7f76f40a9f00 con 0x7f76f4072d30 2024-04-26T02:26:18.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76f8962640 1 -- 172.21.15.87:0/379804153 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76e0003530 con 0x7f76f4072d30 2024-04-26T02:26:18.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76f8962640 1 -- 172.21.15.87:0/379804153 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4129543497 0 0) 0x7f76f40708b0 con 0x7f76f4135cc0 2024-04-26T02:26:18.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76f8962640 1 -- 172.21.15.87:0/379804153 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76f40a9f00 con 0x7f76f4135cc0 2024-04-26T02:26:18.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76f8962640 1 -- 172.21.15.87:0/379804153 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4000472643 0 0) 0x7f76e0003530 con 0x7f76f4072d30 2024-04-26T02:26:18.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76f8962640 1 -- 172.21.15.87:0/379804153 >> v1:172.21.15.87:6790/0 conn(0x7f76f4135cc0 legacy=0x7f76f40701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:26:18.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76f8962640 1 -- 172.21.15.87:0/379804153 >> v1:172.21.15.122:6789/0 conn(0x7f76f41360a0 legacy=0x7f76f4072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:26:18.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76f8962640 1 -- 172.21.15.87:0/379804153 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f40a8e10 con 0x7f76f4072d30 2024-04-26T02:26:18.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76f8962640 1 -- 172.21.15.87:0/379804153 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f76f0001620 con 0x7f76f4072d30 2024-04-26T02:26:18.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76f8962640 1 -- 172.21.15.87:0/379804153 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f76f0001880 con 0x7f76f4072d30 2024-04-26T02:26:18.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76f8962640 1 -- 172.21.15.87:0/379804153 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f76f0001c50 con 0x7f76f4072d30 2024-04-26T02:26:18.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76fbbc8640 1 -- 172.21.15.87:0/379804153 >> v1:172.21.15.87:6789/0 conn(0x7f76f4072d30 legacy=0x7f76f414ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:26:18.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76fbbc8640 1 -- 172.21.15.87:0/379804153 shutdown_connections 2024-04-26T02:26:18.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76fbbc8640 1 -- 172.21.15.87:0/379804153 >> 172.21.15.87:0/379804153 conn(0x7f76f406c490 msgr2=0x7f76f406c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:26:18.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76fbbc8640 1 -- 172.21.15.87:0/379804153 shutdown_connections 2024-04-26T02:26:18.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.092+0000 7f76fbbc8640 1 -- 172.21.15.87:0/379804153 wait complete. 2024-04-26T02:26:18.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76fbbc8640 1 Processor -- start 2024-04-26T02:26:18.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76fbbc8640 1 -- start start 2024-04-26T02:26:18.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76fbbc8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76f412fb90 con 0x7f76f4135cc0 2024-04-26T02:26:18.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76fbbc8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76f4138ab0 con 0x7f76f4072d30 2024-04-26T02:26:18.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76fbbc8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76f413e920 con 0x7f76f41d6f80 2024-04-26T02:26:18.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76f9163640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f76f4135cc0 0x7f76f412ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:34732/0 (socket says 172.21.15.87:34732) 2024-04-26T02:26:18.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76f9163640 1 -- 172.21.15.87:0/1761156034 learned_addr learned my addr 172.21.15.87:0/1761156034 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-04-26T02:26:18.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2940315112 0 0) 0x7f76f412fb90 con 0x7f76f4135cc0 2024-04-26T02:26:18.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76d8003530 con 0x7f76f4135cc0 2024-04-26T02:26:18.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4082669028 0 0) 0x7f76f413e920 con 0x7f76f41d6f80 2024-04-26T02:26:18.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76f412fb90 con 0x7f76f41d6f80 2024-04-26T02:26:18.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4093001102 0 0) 0x7f76d8003530 con 0x7f76f4135cc0 2024-04-26T02:26:18.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f76f413e920 con 0x7f76f4135cc0 2024-04-26T02:26:18.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f76e4001980 con 0x7f76f4135cc0 2024-04-26T02:26:18.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4145554466 0 0) 0x7f76f412fb90 con 0x7f76f41d6f80 2024-04-26T02:26:18.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f76d8003530 con 0x7f76f41d6f80 2024-04-26T02:26:18.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f76f0001bd0 con 0x7f76f41d6f80 2024-04-26T02:26:18.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1148448055 0 0) 0x7f76f4138ab0 con 0x7f76f4072d30 2024-04-26T02:26:18.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76f412fb90 con 0x7f76f4072d30 2024-04-26T02:26:18.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3580139998 0 0) 0x7f76f413e920 con 0x7f76f4135cc0 2024-04-26T02:26:18.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 >> v1:172.21.15.87:6790/0 conn(0x7f76f41d6f80 legacy=0x7f76f412f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:26:18.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 >> v1:172.21.15.122:6789/0 conn(0x7f76f4072d30 legacy=0x7f76f41d6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:26:18.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f413eaf0 con 0x7f76f4135cc0 2024-04-26T02:26:18.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76fbbc8640 1 -- 172.21.15.87:0/1761156034 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f76f41deba0 con 0x7f76f4135cc0 2024-04-26T02:26:18.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76fbbc8640 1 -- 172.21.15.87:0/1761156034 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f76f41def40 con 0x7f76f4135cc0 2024-04-26T02:26:18.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f76e4003950 con 0x7f76f4135cc0 2024-04-26T02:26:18.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1130141913 0 0) 0x7f76e4003b80 con 0x7f76f4135cc0 2024-04-26T02:26:18.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76fbbc8640 1 -- 172.21.15.87:0/1761156034 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76dc005090 con 0x7f76f4135cc0 2024-04-26T02:26:18.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.096+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 5) ==== 87410+0+0 (unknown 1097853133 0 0) 0x7f76e4003da0 con 0x7f76f4135cc0 2024-04-26T02:26:18.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.100+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(24..24 src has 1..24) ==== 4525+0+0 (unknown 3646880037 0 0) 0x7f76e4081410 con 0x7f76f4135cc0 2024-04-26T02:26:18.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.104+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f76e4052280 con 0x7f76f4135cc0 2024-04-26T02:26:18.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.264+0000 7f76fbbc8640 1 -- 172.21.15.87:0/1761156034 --> v1:172.21.15.87:6789/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f76dc005380 con 0x7f76f4135cc0 2024-04-26T02:26:18.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.268+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) ==== 123+0+0 (unknown 1453024670 0 0) 0x7f76e4051b80 con 0x7f76f4135cc0 2024-04-26T02:26:18.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.268+0000 7f76ea7fc640 1 -- 172.21.15.87:0/1761156034 <== mon.0 v1:172.21.15.87:6789/0 11 ==== config(1 keys) ==== 35+0+0 (unknown 2555425225 0 0) 0x7f76e4055f30 con 0x7f76f4135cc0 2024-04-26T02:26:18.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.272+0000 7f76fbbc8640 1 -- 172.21.15.87:0/1761156034 >> v1:172.21.15.122:6816/416193625 conn(0x7f76d8068390 legacy=0x7f76d806a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:26:18.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.272+0000 7f76fbbc8640 1 -- 172.21.15.87:0/1761156034 >> v1:172.21.15.87:6789/0 conn(0x7f76f4135cc0 legacy=0x7f76f412ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:26:18.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.272+0000 7f76fbbc8640 1 -- 172.21.15.87:0/1761156034 shutdown_connections 2024-04-26T02:26:18.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.272+0000 7f76fbbc8640 1 -- 172.21.15.87:0/1761156034 >> 172.21.15.87:0/1761156034 conn(0x7f76f406c490 msgr2=0x7f76f4074c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:26:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.272+0000 7f76fbbc8640 1 -- 172.21.15.87:0/1761156034 shutdown_connections 2024-04-26T02:26:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-26T02:26:18.272+0000 7f76fbbc8640 1 -- 172.21.15.87:0/1761156034 wait complete. 2024-04-26T02:26:18.290 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-26T02:26:18.291 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-26T02:26:18.291 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-26T02:26:18.291 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:26:22.952 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-26T02:26:24.393 INFO:tasks.ceph.osd.0:Stopped 2024-04-26T02:26:24.393 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-26T02:26:24.393 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:26:24.477 INFO:tasks.ceph.osd.1:Stopped 2024-04-26T02:26:24.478 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-26T02:26:24.478 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:26:28.961 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-26T02:26:28.961 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-26T02:26:30.580 INFO:tasks.ceph.osd.2:Stopped 2024-04-26T02:26:30.580 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-26T02:26:30.580 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:26:30.672 INFO:tasks.ceph.osd.3:Stopped 2024-04-26T02:26:30.672 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-26T02:26:30.672 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:26:34.769 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-04-26T02:26:34.769 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-26T02:26:36.774 INFO:tasks.ceph.osd.4:Stopped 2024-04-26T02:26:36.774 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-26T02:26:36.775 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:26:36.864 INFO:tasks.ceph.osd.5:Stopped 2024-04-26T02:26:36.864 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-26T02:26:36.865 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:26:40.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-04-26T02:26:40.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-04-26T02:26:42.966 INFO:tasks.ceph.osd.6:Stopped 2024-04-26T02:26:42.967 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-26T02:26:42.967 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:26:43.050 INFO:tasks.ceph.osd.7:Stopped 2024-04-26T02:26:43.050 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-26T02:26:43.050 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-26T02:26:43.050 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:26:43.079 INFO:tasks.ceph.mgr.y:Stopped 2024-04-26T02:26:43.079 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-26T02:26:43.079 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:26:43.109 INFO:tasks.ceph.mgr.x:Stopped 2024-04-26T02:26:43.109 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-26T02:26:43.109 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T02:26:43.109 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:26:43.136 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T02:26:43.136 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T02:26:43.136 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:26:43.198 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T02:26:43.198 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T02:26:43.198 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T02:26:43.218 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T02:26:43.219 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-26T02:26:43.219 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | head -n 1 2024-04-26T02:26:43.254 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi087.front.sepia.ceph.com 2024-04-26T02:26:43.254 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-26T02:26:43.516 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi087.front.sepia.ceph.com 2024-04-26T02:26:43.516 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-26T02:26:43.697 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi087.front.sepia.ceph.com 2024-04-26T02:26:43.697 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-26T02:26:43.832 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi087.front.sepia.ceph.com 2024-04-26T02:26:43.832 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-26T02:26:44.003 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi122.front.sepia.ceph.com 2024-04-26T02:26:44.004 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-26T02:26:44.236 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi122.front.sepia.ceph.com 2024-04-26T02:26:44.236 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-26T02:26:44.414 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi122.front.sepia.ceph.com 2024-04-26T02:26:44.414 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-26T02:26:44.599 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi122.front.sepia.ceph.com 2024-04-26T02:26:44.600 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-26T02:26:44.791 INFO:tasks.ceph:Archiving mon data... 2024-04-26T02:26:44.792 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674282/data/mon.a.tgz 2024-04-26T02:26:44.792 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-04-26T02:26:44.799 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.DP5KPPoTtf 2024-04-26T02:26:44.799 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.DP5KPPoTtf 2024-04-26T02:26:44.909 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.DP5KPPoTtf 2024-04-26T02:26:44.972 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.DP5KPPoTtf is 308KB 2024-04-26T02:26:45.034 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.DP5KPPoTtf 2024-04-26T02:26:45.039 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674282/data/mon.c.tgz 2024-04-26T02:26:45.040 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-04-26T02:26:45.086 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.hdTwntZ5N5 2024-04-26T02:26:45.086 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.hdTwntZ5N5 2024-04-26T02:26:45.203 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.hdTwntZ5N5 2024-04-26T02:26:45.265 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.hdTwntZ5N5 is 311KB 2024-04-26T02:26:45.323 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.hdTwntZ5N5 2024-04-26T02:26:45.328 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674282/data/mon.b.tgz 2024-04-26T02:26:45.328 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-04-26T02:26:45.334 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.UKiZlU3KU1 2024-04-26T02:26:45.335 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.UKiZlU3KU1 2024-04-26T02:26:45.376 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-04-26T02:26:45.451 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.UKiZlU3KU1 2024-04-26T02:26:45.510 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.UKiZlU3KU1 is 311KB 2024-04-26T02:26:45.569 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.UKiZlU3KU1 2024-04-26T02:26:45.574 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-26T02:26:45.574 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T02:26:45.576 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T02:26:45.636 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T02:26:45.679 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T02:26:45.679 INFO:tasks.ceph:Archiving crash dumps... 2024-04-26T02:26:45.681 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674282/remote/smithi087/crash 2024-04-26T02:26:45.682 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T02:26:45.696 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674282/remote/smithi122/crash 2024-04-26T02:26:45.697 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T02:26:45.734 INFO:tasks.ceph:Compressing logs... 2024-04-26T02:26:45.735 DEBUG:teuthology.orchestra.run.smithi087:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T02:26:45.741 DEBUG:teuthology.orchestra.run.smithi122:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T02:26:45.756 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28892.log 2024-04-26T02:26:45.757 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29491.log 2024-04-26T02:26:45.759 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.28892.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-26T02:26:45.759 INFO:teuthology.orchestra.run.smithi087.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.28892.log.gz 2024-04-26T02:26:45.759 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29567.log 2024-04-26T02:26:45.759 INFO:teuthology.orchestra.run.smithi087.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.29491.log.gz 2024-04-26T02:26:45.759 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25240.log 2024-04-26T02:26:45.759 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29567.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-26T02:26:45.759 INFO:teuthology.orchestra.run.smithi087.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.29567.log.gz 2024-04-26T02:26:45.760 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.25240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29517.log 2024-04-26T02:26:45.760 INFO:teuthology.orchestra.run.smithi087.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.25240.log.gz 2024-04-26T02:26:45.760 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30264.log 2024-04-26T02:26:45.761 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30680.log 2024-04-26T02:26:45.761 INFO:teuthology.orchestra.run.smithi087.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.29517.log.gz 2024-04-26T02:26:45.761 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.30235.log 2024-04-26T02:26:45.761 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30680.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.30264.log.gz 2024-04-26T02:26:45.762 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30342.log 2024-04-26T02:26:45.762 INFO:teuthology.orchestra.run.smithi087.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.30680.log.gz 2024-04-26T02:26:45.762 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29103.log 2024-04-26T02:26:45.762 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.libcephsqlite.30235.log: /var/log/ceph/ceph-client.admin.30342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28878.log 2024-04-26T02:26:45.762 INFO:teuthology.orchestra.run.smithi087.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.30342.log.gz 2024-04-26T02:26:45.763 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29103.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-26T02:26:45.763 INFO:teuthology.orchestra.run.smithi087.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.29103.log.gz 2024-04-26T02:26:45.764 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.28878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29081.log 2024-04-26T02:26:45.764 INFO:teuthology.orchestra.run.smithi087.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.28878.log.gz 2024-04-26T02:26:45.764 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.30049.log 2024-04-26T02:26:45.764 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29081.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-26T02:26:45.764 INFO:teuthology.orchestra.run.smithi087.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.29081.log.gz 2024-04-26T02:26:45.764 INFO:teuthology.orchestra.run.smithi087.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.30235.log.gz 2024-04-26T02:26:45.764 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.libcephsqlite.30049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.30005.log 2024-04-26T02:26:45.772 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29311.log 2024-04-26T02:26:45.772 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.libcephsqlite.30005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30706.log 2024-04-26T02:26:45.772 INFO:teuthology.orchestra.run.smithi087.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30290.log 2024-04-26T02:26:45.772 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29311.log: -- replaced with /var/log/ceph/ceph-client.libcephsqlite.30005.log.gz 2024-04-26T02:26:45.772 INFO:teuthology.orchestra.run.smithi087.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.29311.log.gz 2024-04-26T02:26:45.772 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29293.log 2024-04-26T02:26:45.772 INFO:teuthology.orchestra.run.smithi087.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-26T02:26:45.772 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30290.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28676.log 2024-04-26T02:26:45.772 INFO:teuthology.orchestra.run.smithi087.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.30290.log.gz 2024-04-26T02:26:45.772 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30706.log.gz 2024-04-26T02:26:45.772 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.30169.log 2024-04-26T02:26:45.772 INFO:teuthology.orchestra.run.smithi087.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.29293.log.gz 2024-04-26T02:26:45.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.28676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30368.log 2024-04-26T02:26:45.773 INFO:teuthology.orchestra.run.smithi087.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.28676.log.gz 2024-04-26T02:26:45.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.libcephsqlite.30169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.29962.log 2024-04-26T02:26:45.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29283.log 2024-04-26T02:26:45.773 INFO:teuthology.orchestra.run.smithi087.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.30169.log.gz 2024-04-26T02:26:45.773 INFO:teuthology.orchestra.run.smithi087.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.30368.log.gz/var/log/ceph/ceph-client.libcephsqlite.29962.log: 2024-04-26T02:26:45.773 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29543.log 2024-04-26T02:26:45.773 INFO:teuthology.orchestra.run.smithi087.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.29962.log.gz 2024-04-26T02:26:45.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28884.log 2024-04-26T02:26:45.773 INFO:teuthology.orchestra.run.smithi087.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.29283.log.gz 2024-04-26T02:26:45.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28701.log 2024-04-26T02:26:45.773 INFO:teuthology.orchestra.run.smithi087.stderr: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.29618.log 2024-04-26T02:26:45.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.28884.log: -- replaced with /var/log/ceph/ceph-client.admin.29543.log.gz 2024-04-26T02:26:45.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.28701.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.28884.log.gz 2024-04-26T02:26:45.774 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-26T02:26:45.774 INFO:teuthology.orchestra.run.smithi087.stderr: 86.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29309.log 2024-04-26T02:26:45.774 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28701.log.gz 2024-04-26T02:26:45.774 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.libcephsqlite.29618.log: /var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29099.log 2024-04-26T02:26:45.774 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30446.log 2024-04-26T02:26:45.774 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29309.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.29309.log.gz 2024-04-26T02:26:45.774 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29099.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.29099.log.gz 2024-04-26T02:26:45.777 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30811.log 2024-04-26T02:26:45.777 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25093.log 2024-04-26T02:26:45.778 INFO:teuthology.orchestra.run.smithi087.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.30446.log.gz 2024-04-26T02:26:45.778 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30811.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.30811.log.gz 2024-04-26T02:26:45.778 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30628.log 2024-04-26T02:26:45.778 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.25093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29299.log 2024-04-26T02:26:45.778 INFO:teuthology.orchestra.run.smithi087.stderr: 91.8% 86.8% -- replaced with /var/log/ceph/ceph-client.admin.25093.log.gz 2024-04-26T02:26:45.778 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.libcephsqlite.30049.log.gz 2024-04-26T02:26:45.778 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30628.log: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.30150.log 2024-04-26T02:26:45.778 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30628.log.gz 2024-04-26T02:26:45.778 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.30081.log 2024-04-26T02:26:45.778 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29299.log: /var/log/ceph/ceph-client.libcephsqlite.30150.log: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29087.log 2024-04-26T02:26:45.778 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29299.log.gz 2024-04-26T02:26:45.778 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.libcephsqlite.30081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.29984.log 2024-04-26T02:26:45.778 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29087.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.29087.log.gz 2024-04-26T02:26:45.780 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-26T02:26:45.782 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.libcephsqlite.29984.log: 91.0% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.29984.log.gz 2024-04-26T02:26:45.782 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28751.log 2024-04-26T02:26:45.792 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.1.log: 94.8% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.30081.log.gz 2024-04-26T02:26:45.793 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28841.log 2024-04-26T02:26:45.794 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.28751.log: 95.2% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.30150.log.gz 86.5% -- replaced with /var/log/ceph/ceph-client.admin.28751.log.gz 2024-04-26T02:26:45.794 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-04-26T02:26:45.794 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30654.log 2024-04-26T02:26:45.795 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29102.log 2024-04-26T02:26:45.795 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.28841.log: /var/log/ceph/ceph-client.admin.30654.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.28841.log.gz 2024-04-26T02:26:45.796 INFO:teuthology.orchestra.run.smithi087.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.30654.log.gz 2024-04-26T02:26:45.796 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30316.log 2024-04-26T02:26:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.30192.log 2024-04-26T02:26:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30316.log: /var/log/ceph/ceph-client.admin.29102.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.29102.log.gz 2024-04-26T02:26:45.797 INFO:teuthology.orchestra.run.smithi087.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.30316.log.gz 2024-04-26T02:26:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30784.log 2024-04-26T02:26:45.798 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-26T02:26:45.798 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.libcephsqlite.30192.log: /var/log/ceph/ceph-client.admin.30784.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.30784.log.gz 2024-04-26T02:26:45.800 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30394.log 2024-04-26T02:26:45.802 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.log: 92.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-26T02:26:45.804 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30420.log 2024-04-26T02:26:45.805 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30394.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.30394.log.gz 2024-04-26T02:26:45.805 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29594.log 2024-04-26T02:26:45.806 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30420.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.30420.log.gz 2024-04-26T02:26:45.806 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30524.log 2024-04-26T02:26:45.807 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29594.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.29594.log.gz 2024-04-26T02:26:45.808 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28895.log 2024-04-26T02:26:45.808 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30524.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.30524.log.gz 2024-04-26T02:26:45.809 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25208.log 2024-04-26T02:26:45.809 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25304.log 2024-04-26T02:26:45.809 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.25208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25184.log 2024-04-26T02:26:45.810 INFO:teuthology.orchestra.run.smithi122.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.25208.log.gz 2024-04-26T02:26:45.810 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.25304.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-26T02:26:45.810 INFO:teuthology.orchestra.run.smithi122.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.25304.log.gz 2024-04-26T02:26:45.811 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25160.log 2024-04-26T02:26:45.812 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.25184.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-26T02:26:45.812 INFO:teuthology.orchestra.run.smithi122.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.25184.log.gz 2024-04-26T02:26:45.812 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.25160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25256.log 2024-04-26T02:26:45.812 INFO:teuthology.orchestra.run.smithi122.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.25160.log.gz 2024-04-26T02:26:45.812 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-26T02:26:45.813 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.25256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25232.log 2024-04-26T02:26:45.813 INFO:teuthology.orchestra.run.smithi122.stderr: 86.0%/var/log/ceph/ceph-osd.5.log: -- replaced with /var/log/ceph/ceph-client.admin.25256.log.gz 2024-04-26T02:26:45.813 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-26T02:26:45.814 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-26T02:26:45.814 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.25232.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-26T02:26:45.814 INFO:teuthology.orchestra.run.smithi122.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.25232.log.gz 2024-04-26T02:26:45.815 INFO:teuthology.orchestra.run.smithi122.stderr:gzip/var/log/ceph/ceph.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.25136.log 2024-04-26T02:26:45.815 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-26T02:26:45.816 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.25136.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-26T02:26:45.817 INFO:teuthology.orchestra.run.smithi122.stderr: 88.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-26T02:26:45.817 INFO:teuthology.orchestra.run.smithi122.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.25136.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25280.log 2024-04-26T02:26:45.817 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-26T02:26:45.817 INFO:teuthology.orchestra.run.smithi122.stderr: 86.2%/var/log/ceph/ceph-mon.b.log: -- replaced with /var/log/ceph/ceph.log.gz 2024-04-26T02:26:45.817 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.25280.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.25280.log.gz 2024-04-26T02:26:45.824 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30602.log 2024-04-26T02:26:45.825 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.28895.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.28895.log.gz 2024-04-26T02:26:45.836 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-26T02:26:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30602.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.30602.log.gz 2024-04-26T02:26:45.838 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25350.log 2024-04-26T02:26:45.852 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.30101.log 2024-04-26T02:26:45.853 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.25350.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.25350.log.gz 2024-04-26T02:26:45.860 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29098.log 2024-04-26T02:26:45.861 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.libcephsqlite.30101.log: 87.8% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.30101.log.gz 2024-04-26T02:26:45.873 INFO:teuthology.orchestra.run.smithi122.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-26T02:26:45.876 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30576.log 2024-04-26T02:26:45.877 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29098.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.29098.log.gz 2024-04-26T02:26:45.884 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29080.log 2024-04-26T02:26:45.885 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30576.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.30576.log.gz 2024-04-26T02:26:45.900 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29292.log 2024-04-26T02:26:45.901 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29080.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.29080.log.gz 2024-04-26T02:26:45.912 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-26T02:26:45.913 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29292.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.29292.log.gz 2024-04-26T02:26:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30758.log 2024-04-26T02:26:45.934 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.audit.log: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-26T02:26:45.944 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30498.log 2024-04-26T02:26:45.945 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30758.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.30758.log.gz 2024-04-26T02:26:45.960 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29086.log 2024-04-26T02:26:45.961 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30498.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.30498.log.gz 2024-04-26T02:26:45.973 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30835.log 2024-04-26T02:26:45.973 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29086.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.29086.log.gz 2024-04-26T02:26:45.973 INFO:teuthology.orchestra.run.smithi122.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-26T02:26:45.984 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28891.log 2024-04-26T02:26:45.985 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30835.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.30835.log.gz 2024-04-26T02:26:45.998 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29308.log 2024-04-26T02:26:45.999 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.28891.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.28891.log.gz 2024-04-26T02:26:46.009 INFO:teuthology.orchestra.run.smithi087.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-26T02:26:46.009 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.30026.log 2024-04-26T02:26:46.011 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29308.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.29308.log.gz 2024-04-26T02:26:46.011 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30472.log 2024-04-26T02:26:46.012 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.libcephsqlite.30026.log: 87.8% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.30026.log.gz 2024-04-26T02:26:46.012 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.libcephsqlite.30124.log 2024-04-26T02:26:46.013 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30472.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.30472.log.gz 2024-04-26T02:26:46.013 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28789.log 2024-04-26T02:26:46.028 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.libcephsqlite.30124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28815.log 2024-04-26T02:26:46.029 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.28789.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.28789.log.gz 2024-04-26T02:26:46.044 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20317.log 2024-04-26T02:26:46.045 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.28815.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.28815.log.gz 2024-04-26T02:26:46.045 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30732.log 2024-04-26T02:26:46.046 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.tmp-client.admin.20317.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20317.log.gz 2024-04-26T02:26:46.060 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28889.log 2024-04-26T02:26:46.061 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30732.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.30732.log.gz 2024-04-26T02:26:46.068 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29298.log 2024-04-26T02:26:46.069 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.28889.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.28889.log.gz 2024-04-26T02:26:46.076 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30550.log 2024-04-26T02:26:46.077 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.29298.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.29298.log.gz 2024-04-26T02:26:46.084 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28887.log 2024-04-26T02:26:46.085 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30550.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.30550.log.gz 2024-04-26T02:26:46.091 INFO:teuthology.orchestra.run.smithi087.stderr: 89.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-26T02:26:46.092 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28894.log 2024-04-26T02:26:46.092 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.28887.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.28887.log.gz 2024-04-26T02:26:46.093 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.28894.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.28894.log.gz 2024-04-26T02:26:46.587 INFO:teuthology.orchestra.run.smithi087.stderr: 95.6% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.30124.log.gz 2024-04-26T02:26:46.797 INFO:teuthology.orchestra.run.smithi087.stderr: 95.8% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.30192.log.gz 2024-04-26T02:26:48.497 INFO:teuthology.orchestra.run.smithi087.stderr: 95.2% -- replaced with /var/log/ceph/ceph-client.libcephsqlite.29618.log.gz 2024-04-26T02:26:49.506 INFO:teuthology.orchestra.run.smithi087.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-26T02:26:51.578 INFO:teuthology.orchestra.run.smithi122.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-26T02:26:56.921 INFO:teuthology.orchestra.run.smithi087.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-26T02:26:58.871 INFO:teuthology.orchestra.run.smithi122.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-26T02:27:02.568 INFO:teuthology.orchestra.run.smithi087.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-26T02:27:03.363 INFO:teuthology.orchestra.run.smithi122.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-26T02:27:04.175 INFO:teuthology.orchestra.run.smithi087.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-26T02:27:04.177 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-04-26T02:27:04.177 INFO:teuthology.orchestra.run.smithi087.stderr:real 0m18.433s 2024-04-26T02:27:04.177 INFO:teuthology.orchestra.run.smithi087.stderr:user 0m53.241s 2024-04-26T02:27:04.177 INFO:teuthology.orchestra.run.smithi087.stderr:sys 0m1.797s 2024-04-26T02:27:05.247 INFO:teuthology.orchestra.run.smithi122.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-26T02:27:05.250 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-26T02:27:05.250 INFO:teuthology.orchestra.run.smithi122.stderr:real 0m19.466s 2024-04-26T02:27:05.250 INFO:teuthology.orchestra.run.smithi122.stderr:user 0m54.336s 2024-04-26T02:27:05.250 INFO:teuthology.orchestra.run.smithi122.stderr:sys 0m1.727s 2024-04-26T02:27:05.250 INFO:tasks.ceph:Archiving logs... 2024-04-26T02:27:05.251 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674282/remote/smithi087/log 2024-04-26T02:27:05.252 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T02:27:12.600 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674282/remote/smithi122/log 2024-04-26T02:27:12.601 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T02:27:20.234 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T02:27:20.243 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T02:27:20.243 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T02:27:20.246 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T02:27:20.308 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T02:27:20.317 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T02:27:20.317 DEBUG:teuthology.orchestra.run.smithi087:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T02:27:20.320 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T02:27:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T02:27:20.331 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================== 2024-04-26T02:27:20.331 INFO:teuthology.orchestra.run.smithi087.stdout:+hv01.front.sepi 67.205.162.81 3 u 5 64 377 0.090 -1.780 2.787 2024-04-26T02:27:20.331 INFO:teuthology.orchestra.run.smithi087.stdout:+hv02.front.sepi 63.231.80.2 3 u 63 64 377 0.073 +1.613 2.307 2024-04-26T02:27:20.331 INFO:teuthology.orchestra.run.smithi087.stdout:*hv03.front.sepi 74.6.168.72 3 u 6 64 377 0.096 -1.357 1.678 2024-04-26T02:27:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T02:27:20.333 INFO:teuthology.orchestra.run.smithi122.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T02:27:20.333 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================== 2024-04-26T02:27:20.333 INFO:teuthology.orchestra.run.smithi122.stdout:+hv01.front.sepi 67.205.162.81 3 u 1 64 377 0.064 -4.954 1.616 2024-04-26T02:27:20.333 INFO:teuthology.orchestra.run.smithi122.stdout:*hv02.front.sepi 63.231.80.2 3 u - 64 377 0.045 +1.488 1.776 2024-04-26T02:27:20.333 INFO:teuthology.orchestra.run.smithi122.stdout:+hv03.front.sepi 74.6.168.72 3 u 1 64 377 0.074 -0.920 1.259 2024-04-26T02:27:20.333 INFO:teuthology.orchestra.run.smithi122.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T02:27:20.333 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T02:27:20.342 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T02:27:20.343 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T02:27:20.352 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T02:27:20.361 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T02:27:20.374 INFO:teuthology.task.internal:Duration was 960.618607 seconds 2024-04-26T02:27:20.374 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T02:27:20.382 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T02:27:20.383 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T02:27:20.386 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T02:27:20.424 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T02:27:20.424 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com 2024-04-26T02:27:20.424 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T02:27:20.469 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-04-26T02:27:20.469 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T02:27:20.480 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T02:27:20.480 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T02:27:20.514 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T02:27:20.552 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T02:27:20.552 DEBUG:teuthology.orchestra.run.smithi087:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T02:27:20.581 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T02:27:20.649 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T02:27:20.659 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T02:27:20.660 DEBUG:teuthology.orchestra.run.smithi087:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T02:27:20.671 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T02:27:20.684 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T02:27:20.693 DEBUG:teuthology.orchestra.run.smithi087:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:27:20.717 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:27:20.724 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = core 2024-04-26T02:27:20.735 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-04-26T02:27:20.759 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:27:20.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:27:20.784 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:27:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:27:20.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T02:27:20.817 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T02:27:20.817 DEBUG:teuthology.misc:Transferring archived files from smithi087:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674282/remote/smithi087 2024-04-26T02:27:20.818 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T02:27:20.845 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674282/remote/smithi122 2024-04-26T02:27:20.846 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T02:27:20.867 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T02:27:20.867 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T02:27:20.885 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T02:27:20.910 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T02:27:20.923 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T02:27:20.923 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T02:27:20.937 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T02:27:20.937 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T02:27:20.939 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T02:27:20.943 INFO:teuthology.orchestra.run.smithi087.stdout: 658751 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 02:27 /home/ubuntu/cephtest 2024-04-26T02:27:20.959 INFO:teuthology.orchestra.run.smithi122.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 02:27 /home/ubuntu/cephtest 2024-04-26T02:27:20.960 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T02:27:20.970 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T02:27:20.994 INFO:teuthology.run:Summary data: description: rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} duration: 960.6186068058014 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-04-26T02:27:20.994 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T02:27:21.074 INFO:teuthology.run:pass