2023-11-16T01:36:27.469 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-16T01:36:27.470 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694 --description rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_workunits} --verbose --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694/orig.config.yaml 2023-11-16T01:36:27.503 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-16T01:36:27.589 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694 branch: wip-yuri3-testing-2023-11-14-1227-pacific description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_workunits} email: yweinste@redhat.com first_in_suite: false job_id: '7459694' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri3-testing-2023-11-14-1227-pacific ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - but it is still running - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(AUTH_BAD_CAPS\) sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri3-testing-2023-11-14-1227-pacific sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 7739 sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri3-testing-2023-11-14-1227-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac targets: smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 20 thrash_delay: 1 - workunit: clients: client.0: - mon/pool_ops.sh - mon/crush_ops.sh - mon/osd.sh - mon/caps.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-15_21:10:46 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1224587 2023-11-16T01:36:27.590 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa; will attempt to use it 2023-11-16T01:36:27.590 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks 2023-11-16T01:36:27.591 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-16T01:36:27.593 INFO:teuthology.task.internal:Checking packages... 2023-11-16T01:36:27.618 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac' 2023-11-16T01:36:27.618 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-16T01:36:27.618 INFO:teuthology.packaging:ref: None 2023-11-16T01:36:27.618 INFO:teuthology.packaging:tag: None 2023-11-16T01:36:27.618 INFO:teuthology.packaging:branch: wip-yuri3-testing-2023-11-14-1227-pacific 2023-11-16T01:36:27.619 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-16T01:36:27.619 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri3-testing-2023-11-14-1227-pacific 2023-11-16T01:36:27.841 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-171-ged3c8e92-1focal 2023-11-16T01:36:27.841 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-16T01:36:27.847 INFO:teuthology.task.internal:no buildpackages task found 2023-11-16T01:36:27.847 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-16T01:36:27.892 INFO:teuthology.task.internal:Saving configuration 2023-11-16T01:36:27.908 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-16T01:36:27.925 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T01:36:27.950 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 01:30:26.791767', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-11-16T01:36:27.973 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 01:30:26.793190', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-11-16T01:36:27.974 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-16T01:36:27.980 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2023-11-16T01:36:27.981 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2023-11-16T01:36:27.981 INFO:teuthology.run_tasks:Running task console_log... 2023-11-16T01:36:28.086 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9987de1f70>, signals=[15]) 2023-11-16T01:36:28.087 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-16T01:36:28.093 INFO:teuthology.task.internal:Opening connections... 2023-11-16T01:36:28.094 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2023-11-16T01:36:28.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T01:36:28.144 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2023-11-16T01:36:28.145 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T01:36:28.194 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-16T01:36:28.200 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2023-11-16T01:36:28.225 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2023-11-16T01:36:28.226 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2023-11-16T01:36:28.272 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="Ubuntu" 2023-11-16T01:36:28.272 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-11-16T01:36:28.272 INFO:teuthology.orchestra.run.smithi083.stdout:ID=ubuntu 2023-11-16T01:36:28.272 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE=debian 2023-11-16T01:36:28.273 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-11-16T01:36:28.273 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="20.04" 2023-11-16T01:36:28.273 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-16T01:36:28.273 INFO:teuthology.orchestra.run.smithi083.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-16T01:36:28.273 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-16T01:36:28.273 INFO:teuthology.orchestra.run.smithi083.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-16T01:36:28.273 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_CODENAME=focal 2023-11-16T01:36:28.273 INFO:teuthology.orchestra.run.smithi083.stdout:UBUNTU_CODENAME=focal 2023-11-16T01:36:28.274 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2023-11-16T01:36:28.292 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2023-11-16T01:36:28.299 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2023-11-16T01:36:28.299 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2023-11-16T01:36:28.305 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="Ubuntu" 2023-11-16T01:36:28.305 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-11-16T01:36:28.305 INFO:teuthology.orchestra.run.smithi136.stdout:ID=ubuntu 2023-11-16T01:36:28.305 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE=debian 2023-11-16T01:36:28.305 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-11-16T01:36:28.305 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="20.04" 2023-11-16T01:36:28.306 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-16T01:36:28.306 INFO:teuthology.orchestra.run.smithi136.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-16T01:36:28.306 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-16T01:36:28.306 INFO:teuthology.orchestra.run.smithi136.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-16T01:36:28.306 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_CODENAME=focal 2023-11-16T01:36:28.306 INFO:teuthology.orchestra.run.smithi136.stdout:UBUNTU_CODENAME=focal 2023-11-16T01:36:28.306 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2023-11-16T01:36:28.325 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-16T01:36:28.334 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-16T01:36:28.367 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-16T01:36:28.367 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2023-11-16T01:36:28.369 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2023-11-16T01:36:28.375 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-16T01:36:28.400 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-16T01:36:28.401 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2023-11-16T01:36:28.403 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2023-11-16T01:36:28.420 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-16T01:36:28.479 INFO:teuthology.run_tasks:Running task kernel... 2023-11-16T01:36:28.493 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-16T01:36:28.494 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-16T01:36:28.494 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T01:36:28.494 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T01:36:28.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:36:28.500 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2023-11-16T01:36:28.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:36:28.501 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2023-11-16T01:36:28.507 INFO:teuthology.orchestra.run.smithi083.stdout:5.4.0-124-generic 2023-11-16T01:36:28.507 INFO:teuthology.task.kernel:Running kernel on smithi083: 5.4.0-124-generic 2023-11-16T01:36:28.507 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-get clean 2023-11-16T01:36:28.508 INFO:teuthology.orchestra.run.smithi136.stdout:5.4.0-124-generic 2023-11-16T01:36:28.508 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.4.0-124-generic 2023-11-16T01:36:28.508 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get clean 2023-11-16T01:36:28.638 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-get update 2023-11-16T01:36:28.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2023-11-16T01:36:28.806 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-16T01:36:28.811 INFO:teuthology.orchestra.run.smithi083.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-11-16T01:36:28.811 INFO:teuthology.orchestra.run.smithi083.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-11-16T01:36:28.819 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-16T01:36:28.822 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-11-16T01:36:28.823 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-11-16T01:36:28.890 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-11-16T01:36:28.891 INFO:teuthology.orchestra.run.smithi083.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-11-16T01:36:29.180 INFO:teuthology.orchestra.run.smithi083.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,569 kB] 2023-11-16T01:36:29.190 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [906 kB] 2023-11-16T01:36:29.283 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,960 kB] 2023-11-16T01:36:29.320 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [481 kB] 2023-11-16T01:36:29.323 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2023-11-16T01:36:29.324 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [36.0 kB] 2023-11-16T01:36:29.324 INFO:teuthology.orchestra.run.smithi136.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,479 kB] 2023-11-16T01:36:29.333 INFO:teuthology.orchestra.run.smithi136.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [674 kB] 2023-11-16T01:36:29.335 INFO:teuthology.orchestra.run.smithi083.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [674 kB] 2023-11-16T01:36:29.337 INFO:teuthology.orchestra.run.smithi083.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [906 kB] 2023-11-16T01:36:29.343 INFO:teuthology.orchestra.run.smithi083.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [398 kB] 2023-11-16T01:36:29.344 INFO:teuthology.orchestra.run.smithi136.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [346 kB] 2023-11-16T01:36:29.347 INFO:teuthology.orchestra.run.smithi136.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B] 2023-11-16T01:36:29.347 INFO:teuthology.orchestra.run.smithi136.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [754 kB] 2023-11-16T01:36:29.348 INFO:teuthology.orchestra.run.smithi083.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-11-16T01:36:29.348 INFO:teuthology.orchestra.run.smithi083.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [34.7 kB] 2023-11-16T01:36:29.348 INFO:teuthology.orchestra.run.smithi083.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,367 kB] 2023-11-16T01:36:29.354 INFO:teuthology.orchestra.run.smithi136.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,129 kB] 2023-11-16T01:36:29.364 INFO:teuthology.orchestra.run.smithi136.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [270 kB] 2023-11-16T01:36:29.366 INFO:teuthology.orchestra.run.smithi136.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2023-11-16T01:36:29.367 INFO:teuthology.orchestra.run.smithi136.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2023-11-16T01:36:29.367 INFO:teuthology.orchestra.run.smithi136.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-11-16T01:36:29.367 INFO:teuthology.orchestra.run.smithi136.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-11-16T01:36:29.372 INFO:teuthology.orchestra.run.smithi136.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-11-16T01:36:29.374 INFO:teuthology.orchestra.run.smithi083.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [329 kB] 2023-11-16T01:36:29.378 INFO:teuthology.orchestra.run.smithi083.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2023-11-16T01:36:29.378 INFO:teuthology.orchestra.run.smithi083.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [621 kB] 2023-11-16T01:36:29.385 INFO:teuthology.orchestra.run.smithi083.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [898 kB] 2023-11-16T01:36:29.395 INFO:teuthology.orchestra.run.smithi083.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [188 kB] 2023-11-16T01:36:29.397 INFO:teuthology.orchestra.run.smithi083.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2023-11-16T01:36:29.397 INFO:teuthology.orchestra.run.smithi083.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-11-16T01:36:29.398 INFO:teuthology.orchestra.run.smithi083.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,188 B] 2023-11-16T01:36:29.398 INFO:teuthology.orchestra.run.smithi083.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-11-16T01:36:29.398 INFO:teuthology.orchestra.run.smithi083.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-11-16T01:36:29.414 INFO:teuthology.orchestra.run.smithi136.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,569 kB] 2023-11-16T01:36:29.418 INFO:teuthology.orchestra.run.smithi083.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,960 kB] 2023-11-16T01:36:29.444 INFO:teuthology.orchestra.run.smithi136.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-11-16T01:36:29.463 INFO:teuthology.orchestra.run.smithi136.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-11-16T01:36:29.464 INFO:teuthology.orchestra.run.smithi136.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-11-16T01:36:29.465 INFO:teuthology.orchestra.run.smithi136.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-11-16T01:36:29.465 INFO:teuthology.orchestra.run.smithi136.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-11-16T01:36:29.466 INFO:teuthology.orchestra.run.smithi136.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-11-16T01:36:29.518 INFO:teuthology.orchestra.run.smithi083.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [481 kB] 2023-11-16T01:36:29.523 INFO:teuthology.orchestra.run.smithi136.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [398 kB] 2023-11-16T01:36:29.529 INFO:teuthology.orchestra.run.smithi083.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2023-11-16T01:36:29.529 INFO:teuthology.orchestra.run.smithi083.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,479 kB] 2023-11-16T01:36:29.532 INFO:teuthology.orchestra.run.smithi136.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-11-16T01:36:29.534 INFO:teuthology.orchestra.run.smithi136.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,367 kB] 2023-11-16T01:36:29.677 INFO:teuthology.orchestra.run.smithi136.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [34.7 kB] 2023-11-16T01:36:29.677 INFO:teuthology.orchestra.run.smithi136.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [329 kB] 2023-11-16T01:36:29.679 INFO:teuthology.orchestra.run.smithi136.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2023-11-16T01:36:29.679 INFO:teuthology.orchestra.run.smithi136.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [621 kB] 2023-11-16T01:36:29.684 INFO:teuthology.orchestra.run.smithi083.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [36.0 kB] 2023-11-16T01:36:29.684 INFO:teuthology.orchestra.run.smithi136.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [898 kB] 2023-11-16T01:36:29.684 INFO:teuthology.orchestra.run.smithi083.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [346 kB] 2023-11-16T01:36:29.686 INFO:teuthology.orchestra.run.smithi083.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B] 2023-11-16T01:36:29.687 INFO:teuthology.orchestra.run.smithi083.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [754 kB] 2023-11-16T01:36:29.692 INFO:teuthology.orchestra.run.smithi083.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,129 kB] 2023-11-16T01:36:29.706 INFO:teuthology.orchestra.run.smithi136.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [188 kB] 2023-11-16T01:36:29.708 INFO:teuthology.orchestra.run.smithi136.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2023-11-16T01:36:29.709 INFO:teuthology.orchestra.run.smithi136.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-11-16T01:36:29.710 INFO:teuthology.orchestra.run.smithi136.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,188 B] 2023-11-16T01:36:29.721 INFO:teuthology.orchestra.run.smithi083.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [270 kB] 2023-11-16T01:36:29.725 INFO:teuthology.orchestra.run.smithi083.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2023-11-16T01:36:29.726 INFO:teuthology.orchestra.run.smithi083.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2023-11-16T01:36:29.727 INFO:teuthology.orchestra.run.smithi083.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-11-16T01:36:29.727 INFO:teuthology.orchestra.run.smithi083.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-11-16T01:36:29.727 INFO:teuthology.orchestra.run.smithi083.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-11-16T01:36:29.749 INFO:teuthology.orchestra.run.smithi083.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-11-16T01:36:29.751 INFO:teuthology.orchestra.run.smithi083.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-11-16T01:36:29.752 INFO:teuthology.orchestra.run.smithi083.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-11-16T01:36:29.752 INFO:teuthology.orchestra.run.smithi083.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-11-16T01:36:29.752 INFO:teuthology.orchestra.run.smithi083.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-11-16T01:36:29.753 INFO:teuthology.orchestra.run.smithi083.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-11-16T01:36:29.760 INFO:teuthology.orchestra.run.smithi136.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-11-16T01:36:29.761 INFO:teuthology.orchestra.run.smithi136.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-11-16T01:36:33.326 INFO:teuthology.orchestra.run.smithi083.stdout:Fetched 18.1 MB in 3s (6,902 kB/s) 2023-11-16T01:36:33.380 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 18.1 MB in 3s (7,099 kB/s) 2023-11-16T01:36:34.731 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2023-11-16T01:36:34.755 DEBUG:teuthology.orchestra.run.smithi083:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-16T01:36:34.780 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2023-11-16T01:36:34.807 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-16T01:36:34.830 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2023-11-16T01:36:34.883 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2023-11-16T01:36:35.034 INFO:teuthology.orchestra.run.smithi083.stdout:Building dependency tree... 2023-11-16T01:36:35.034 INFO:teuthology.orchestra.run.smithi083.stdout:Reading state information... 2023-11-16T01:36:35.087 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2023-11-16T01:36:35.088 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2023-11-16T01:36:35.220 INFO:teuthology.orchestra.run.smithi083.stdout:The following additional packages will be installed: 2023-11-16T01:36:35.220 INFO:teuthology.orchestra.run.smithi083.stdout: linux-generic linux-headers-5.4.0-166 linux-headers-5.4.0-166-generic 2023-11-16T01:36:35.220 INFO:teuthology.orchestra.run.smithi083.stdout: linux-headers-generic linux-image-5.4.0-166-generic 2023-11-16T01:36:35.220 INFO:teuthology.orchestra.run.smithi083.stdout: linux-modules-5.4.0-166-generic linux-modules-extra-5.4.0-166-generic 2023-11-16T01:36:35.221 INFO:teuthology.orchestra.run.smithi083.stdout:Suggested packages: 2023-11-16T01:36:35.221 INFO:teuthology.orchestra.run.smithi083.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-11-16T01:36:35.221 INFO:teuthology.orchestra.run.smithi083.stdout:Recommended packages: 2023-11-16T01:36:35.221 INFO:teuthology.orchestra.run.smithi083.stdout: thermald 2023-11-16T01:36:35.271 INFO:teuthology.orchestra.run.smithi083.stdout:The following NEW packages will be installed: 2023-11-16T01:36:35.271 INFO:teuthology.orchestra.run.smithi083.stdout: linux-headers-5.4.0-166 linux-headers-5.4.0-166-generic 2023-11-16T01:36:35.271 INFO:teuthology.orchestra.run.smithi083.stdout: linux-image-5.4.0-166-generic linux-modules-5.4.0-166-generic 2023-11-16T01:36:35.272 INFO:teuthology.orchestra.run.smithi083.stdout: linux-modules-extra-5.4.0-166-generic 2023-11-16T01:36:35.272 INFO:teuthology.orchestra.run.smithi136.stdout:The following additional packages will be installed: 2023-11-16T01:36:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: linux-generic linux-headers-5.4.0-166 linux-headers-5.4.0-166-generic 2023-11-16T01:36:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: linux-headers-generic linux-image-5.4.0-166-generic 2023-11-16T01:36:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: linux-modules-5.4.0-166-generic linux-modules-extra-5.4.0-166-generic 2023-11-16T01:36:35.273 INFO:teuthology.orchestra.run.smithi136.stdout:Suggested packages: 2023-11-16T01:36:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-11-16T01:36:35.273 INFO:teuthology.orchestra.run.smithi136.stdout:Recommended packages: 2023-11-16T01:36:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: thermald 2023-11-16T01:36:35.273 INFO:teuthology.orchestra.run.smithi083.stdout:The following packages will be upgraded: 2023-11-16T01:36:35.273 INFO:teuthology.orchestra.run.smithi083.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-16T01:36:35.331 INFO:teuthology.orchestra.run.smithi136.stdout:The following NEW packages will be installed: 2023-11-16T01:36:35.332 INFO:teuthology.orchestra.run.smithi136.stdout: linux-headers-5.4.0-166 linux-headers-5.4.0-166-generic 2023-11-16T01:36:35.332 INFO:teuthology.orchestra.run.smithi136.stdout: linux-image-5.4.0-166-generic linux-modules-5.4.0-166-generic 2023-11-16T01:36:35.332 INFO:teuthology.orchestra.run.smithi136.stdout: linux-modules-extra-5.4.0-166-generic 2023-11-16T01:36:35.333 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages will be upgraded: 2023-11-16T01:36:35.333 INFO:teuthology.orchestra.run.smithi136.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-16T01:36:35.514 INFO:teuthology.orchestra.run.smithi136.stdout:3 upgraded, 5 newly installed, 0 to remove and 278 not upgraded. 2023-11-16T01:36:35.514 INFO:teuthology.orchestra.run.smithi136.stdout:Need to get 77.1 MB of archives. 2023-11-16T01:36:35.515 INFO:teuthology.orchestra.run.smithi136.stdout:After this operation, 380 MB of additional disk space will be used. 2023-11-16T01:36:35.515 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-166-generic amd64 5.4.0-166.183 [15.0 MB] 2023-11-16T01:36:36.670 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-166-generic amd64 5.4.0-166.183 [10.5 MB] 2023-11-16T01:36:36.942 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-166-generic amd64 5.4.0-166.183 [39.2 MB] 2023-11-16T01:36:38.245 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.166.163 [1,900 B] 2023-11-16T01:36:38.246 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.166.163 [2,592 B] 2023-11-16T01:36:38.246 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-166 all 5.4.0-166.183 [11.0 MB] 2023-11-16T01:36:38.527 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-166-generic amd64 5.4.0-166.183 [1,370 kB] 2023-11-16T01:36:38.556 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.166.163 [2,464 B] 2023-11-16T01:36:38.576 INFO:teuthology.orchestra.run.smithi083.stdout:3 upgraded, 5 newly installed, 0 to remove and 278 not upgraded. 2023-11-16T01:36:38.576 INFO:teuthology.orchestra.run.smithi083.stdout:Need to get 77.1 MB of archives. 2023-11-16T01:36:38.576 INFO:teuthology.orchestra.run.smithi083.stdout:After this operation, 380 MB of additional disk space will be used. 2023-11-16T01:36:38.576 INFO:teuthology.orchestra.run.smithi083.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-166-generic amd64 5.4.0-166.183 [15.0 MB] 2023-11-16T01:36:38.905 INFO:teuthology.orchestra.run.smithi083.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-166-generic amd64 5.4.0-166.183 [10.5 MB] 2023-11-16T01:36:38.998 INFO:teuthology.orchestra.run.smithi083.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-166-generic amd64 5.4.0-166.183 [39.2 MB] 2023-11-16T01:36:39.230 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 77.1 MB in 3s (24.0 MB/s) 2023-11-16T01:36:39.398 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-modules-5.4.0-166-generic. 2023-11-16T01:36:39.553 INFO:teuthology.orchestra.run.smithi083.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.166.163 [1,900 B] 2023-11-16T01:36:39.553 INFO:teuthology.orchestra.run.smithi083.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.166.163 [2,592 B] 2023-11-16T01:36:39.553 INFO:teuthology.orchestra.run.smithi083.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-166 all 5.4.0-166.183 [11.0 MB] 2023-11-16T01:36:39.651 INFO:teuthology.orchestra.run.smithi083.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-166-generic amd64 5.4.0-166.183 [1,370 kB] 2023-11-16T01:36:39.663 INFO:teuthology.orchestra.run.smithi083.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.166.163 [2,464 B] 2023-11-16T01:36:40.387 INFO:teuthology.orchestra.run.smithi083.stdout:Fetched 77.1 MB in 4s (17.6 MB/s) 2023-11-16T01:36:40.589 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package linux-modules-5.4.0-166-generic. 2023-11-16T01:36:41.101 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-11-16T01:36:41.106 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../0-linux-modules-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-16T01:36:41.186 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-modules-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:36:42.289 INFO:teuthology.orchestra.run.smithi083.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-11-16T01:36:42.294 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../0-linux-modules-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-16T01:36:42.426 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-modules-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:36:43.127 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-image-5.4.0-166-generic. 2023-11-16T01:36:43.140 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../1-linux-image-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-16T01:36:43.239 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-image-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:36:44.280 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-166-generic. 2023-11-16T01:36:44.296 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-16T01:36:44.338 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-modules-extra-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:36:44.351 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package linux-image-5.4.0-166-generic. 2023-11-16T01:36:44.366 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../1-linux-image-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-16T01:36:44.456 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-image-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:36:45.555 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-166-generic. 2023-11-16T01:36:45.568 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-16T01:36:45.621 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-modules-extra-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:36:49.427 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../3-linux-generic_5.4.0.166.163_amd64.deb ... 2023-11-16T01:36:49.569 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-generic (5.4.0.166.163) over (5.4.0.124.125) ... 2023-11-16T01:36:49.911 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.166.163_amd64.deb ... 2023-11-16T01:36:50.039 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-image-generic (5.4.0.166.163) over (5.4.0.124.125) ... 2023-11-16T01:36:50.384 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-headers-5.4.0-166. 2023-11-16T01:36:50.399 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../5-linux-headers-5.4.0-166_5.4.0-166.183_all.deb ... 2023-11-16T01:36:50.428 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../3-linux-generic_5.4.0.166.163_amd64.deb ... 2023-11-16T01:36:50.442 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-headers-5.4.0-166 (5.4.0-166.183) ... 2023-11-16T01:36:50.551 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-generic (5.4.0.166.163) over (5.4.0.124.125) ... 2023-11-16T01:36:50.921 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.166.163_amd64.deb ... 2023-11-16T01:36:51.055 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-image-generic (5.4.0.166.163) over (5.4.0.124.125) ... 2023-11-16T01:36:51.399 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package linux-headers-5.4.0-166. 2023-11-16T01:36:51.413 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../5-linux-headers-5.4.0-166_5.4.0-166.183_all.deb ... 2023-11-16T01:36:51.441 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-headers-5.4.0-166 (5.4.0-166.183) ... 2023-11-16T01:36:55.955 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-headers-5.4.0-166-generic. 2023-11-16T01:36:55.978 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../6-linux-headers-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-16T01:36:56.021 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-headers-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:36:57.159 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package linux-headers-5.4.0-166-generic. 2023-11-16T01:36:57.180 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../6-linux-headers-5.4.0-166-generic_5.4.0-166.183_amd64.deb ... 2023-11-16T01:36:57.217 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-headers-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:36:58.667 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.166.163_amd64.deb ... 2023-11-16T01:36:58.796 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-headers-generic (5.4.0.166.163) over (5.4.0.124.125) ... 2023-11-16T01:36:59.115 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-modules-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:37:00.064 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.166.163_amd64.deb ... 2023-11-16T01:37:00.191 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-headers-generic (5.4.0.166.163) over (5.4.0.124.125) ... 2023-11-16T01:37:00.360 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-headers-5.4.0-166 (5.4.0-166.183) ... 2023-11-16T01:37:00.477 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-image-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:37:00.525 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-modules-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:37:01.627 INFO:teuthology.orchestra.run.smithi136.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-166-generic 2023-11-16T01:37:01.627 INFO:teuthology.orchestra.run.smithi136.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-166-generic 2023-11-16T01:37:01.762 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-headers-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:37:01.781 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-headers-5.4.0-166 (5.4.0-166.183) ... 2023-11-16T01:37:01.888 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-modules-extra-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:37:01.907 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-image-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:37:03.073 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-headers-generic (5.4.0.166.163) ... 2023-11-16T01:37:03.199 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-image-generic (5.4.0.166.163) ... 2023-11-16T01:37:03.333 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-generic (5.4.0.166.163) ... 2023-11-16T01:37:03.338 INFO:teuthology.orchestra.run.smithi083.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-166-generic 2023-11-16T01:37:03.338 INFO:teuthology.orchestra.run.smithi083.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-166-generic 2023-11-16T01:37:03.468 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for linux-image-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:37:03.536 INFO:teuthology.orchestra.run.smithi136.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-16T01:37:03.536 INFO:teuthology.orchestra.run.smithi136.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-166-generic 2023-11-16T01:37:04.101 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-headers-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:37:04.244 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-modules-extra-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:37:05.420 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-headers-generic (5.4.0.166.163) ... 2023-11-16T01:37:05.530 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-image-generic (5.4.0.166.163) ... 2023-11-16T01:37:05.673 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-generic (5.4.0.166.163) ... 2023-11-16T01:37:05.799 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for linux-image-5.4.0-166-generic (5.4.0-166.183) ... 2023-11-16T01:37:05.865 INFO:teuthology.orchestra.run.smithi083.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-16T01:37:05.865 INFO:teuthology.orchestra.run.smithi083.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-166-generic 2023-11-16T01:37:26.201 INFO:teuthology.orchestra.run.smithi136.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-16T01:37:26.202 INFO:teuthology.orchestra.run.smithi136.stdout:Sourcing file `/etc/default/grub' 2023-11-16T01:37:26.206 INFO:teuthology.orchestra.run.smithi136.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-16T01:37:26.228 INFO:teuthology.orchestra.run.smithi136.stdout:Generating grub configuration file ... 2023-11-16T01:37:26.564 INFO:teuthology.orchestra.run.smithi136.stdout:Found linux image: /boot/vmlinuz-5.4.0-166-generic 2023-11-16T01:37:26.591 INFO:teuthology.orchestra.run.smithi136.stdout:Found initrd image: /boot/initrd.img-5.4.0-166-generic 2023-11-16T01:37:26.966 INFO:teuthology.orchestra.run.smithi136.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-11-16T01:37:26.977 INFO:teuthology.orchestra.run.smithi136.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-11-16T01:37:27.807 INFO:teuthology.orchestra.run.smithi083.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-16T01:37:27.807 INFO:teuthology.orchestra.run.smithi083.stdout:Sourcing file `/etc/default/grub' 2023-11-16T01:37:27.828 INFO:teuthology.orchestra.run.smithi083.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-16T01:37:27.857 INFO:teuthology.orchestra.run.smithi083.stdout:Generating grub configuration file ... 2023-11-16T01:37:28.219 INFO:teuthology.orchestra.run.smithi083.stdout:Found linux image: /boot/vmlinuz-5.4.0-166-generic 2023-11-16T01:37:28.243 INFO:teuthology.orchestra.run.smithi083.stdout:Found initrd image: /boot/initrd.img-5.4.0-166-generic 2023-11-16T01:37:28.656 INFO:teuthology.orchestra.run.smithi083.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-11-16T01:37:28.670 INFO:teuthology.orchestra.run.smithi083.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-11-16T01:37:28.728 INFO:teuthology.orchestra.run.smithi136.stdout:done 2023-11-16T01:37:29.152 DEBUG:teuthology.orchestra.run.smithi136:> dpkg -s linux-image-generic 2023-11-16T01:37:29.207 INFO:teuthology.orchestra.run.smithi136.stdout:Package: linux-image-generic 2023-11-16T01:37:29.207 INFO:teuthology.orchestra.run.smithi136.stdout:Status: install ok installed 2023-11-16T01:37:29.208 INFO:teuthology.orchestra.run.smithi136.stdout:Priority: optional 2023-11-16T01:37:29.208 INFO:teuthology.orchestra.run.smithi136.stdout:Section: kernel 2023-11-16T01:37:29.208 INFO:teuthology.orchestra.run.smithi136.stdout:Installed-Size: 20 2023-11-16T01:37:29.208 INFO:teuthology.orchestra.run.smithi136.stdout:Maintainer: Ubuntu Kernel Team 2023-11-16T01:37:29.208 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: amd64 2023-11-16T01:37:29.208 INFO:teuthology.orchestra.run.smithi136.stdout:Source: linux-meta 2023-11-16T01:37:29.208 INFO:teuthology.orchestra.run.smithi136.stdout:Version: 5.4.0.166.163 2023-11-16T01:37:29.208 INFO:teuthology.orchestra.run.smithi136.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-11-16T01:37:29.208 INFO:teuthology.orchestra.run.smithi136.stdout:Depends: linux-image-5.4.0-166-generic, linux-modules-extra-5.4.0-166-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-16T01:37:29.208 INFO:teuthology.orchestra.run.smithi136.stdout:Recommends: thermald 2023-11-16T01:37:29.208 INFO:teuthology.orchestra.run.smithi136.stdout:Description: Generic Linux kernel image 2023-11-16T01:37:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: This package will always depend on the latest generic kernel image 2023-11-16T01:37:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: available. 2023-11-16T01:37:29.209 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-166-generic 2023-11-16T01:37:29.209 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-16T01:37:29.209 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-16T01:37:29.209 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-11-16T01:37:29.209 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi136.front.sepia.ceph.com, path=None, version=distro) 2023-11-16T01:37:29.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get clean 2023-11-16T01:37:29.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2023-11-16T01:37:29.418 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-11-16T01:37:29.536 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-16T01:37:29.618 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-11-16T01:37:29.708 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-11-16T01:37:30.444 INFO:teuthology.orchestra.run.smithi083.stdout:done 2023-11-16T01:37:30.858 DEBUG:teuthology.orchestra.run.smithi083:> dpkg -s linux-image-generic 2023-11-16T01:37:30.908 INFO:teuthology.orchestra.run.smithi083.stdout:Package: linux-image-generic 2023-11-16T01:37:30.909 INFO:teuthology.orchestra.run.smithi083.stdout:Status: install ok installed 2023-11-16T01:37:30.909 INFO:teuthology.orchestra.run.smithi083.stdout:Priority: optional 2023-11-16T01:37:30.909 INFO:teuthology.orchestra.run.smithi083.stdout:Section: kernel 2023-11-16T01:37:30.909 INFO:teuthology.orchestra.run.smithi083.stdout:Installed-Size: 20 2023-11-16T01:37:30.909 INFO:teuthology.orchestra.run.smithi083.stdout:Maintainer: Ubuntu Kernel Team 2023-11-16T01:37:30.909 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: amd64 2023-11-16T01:37:30.909 INFO:teuthology.orchestra.run.smithi083.stdout:Source: linux-meta 2023-11-16T01:37:30.909 INFO:teuthology.orchestra.run.smithi083.stdout:Version: 5.4.0.166.163 2023-11-16T01:37:30.909 INFO:teuthology.orchestra.run.smithi083.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-11-16T01:37:30.909 INFO:teuthology.orchestra.run.smithi083.stdout:Depends: linux-image-5.4.0-166-generic, linux-modules-extra-5.4.0-166-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-16T01:37:30.909 INFO:teuthology.orchestra.run.smithi083.stdout:Recommends: thermald 2023-11-16T01:37:30.910 INFO:teuthology.orchestra.run.smithi083.stdout:Description: Generic Linux kernel image 2023-11-16T01:37:30.910 INFO:teuthology.orchestra.run.smithi083.stdout: This package will always depend on the latest generic kernel image 2023-11-16T01:37:30.910 INFO:teuthology.orchestra.run.smithi083.stdout: available. 2023-11-16T01:37:30.910 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-166-generic 2023-11-16T01:37:30.910 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-16T01:37:30.910 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-16T01:37:30.910 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-11-16T01:37:30.910 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi083.front.sepia.ceph.com, path=None, version=distro) 2023-11-16T01:37:30.910 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-get clean 2023-11-16T01:37:30.992 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-get update 2023-11-16T01:37:31.163 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2023-11-16T01:37:31.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-16T01:37:31.249 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2023-11-16T01:37:31.256 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-16T01:37:31.257 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-11-16T01:37:31.327 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-11-16T01:37:31.413 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-11-16T01:37:31.462 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2023-11-16T01:37:31.463 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2023-11-16T01:37:31.683 INFO:teuthology.orchestra.run.smithi136.stdout:linux-image-generic is already the newest version (5.4.0.166.163). 2023-11-16T01:37:31.684 INFO:teuthology.orchestra.run.smithi136.stdout:0 upgraded, 0 newly installed, 0 to remove and 278 not upgraded. 2023-11-16T01:37:31.686 DEBUG:teuthology.orchestra.run.smithi136:> dpkg -s linux-image-generic 2023-11-16T01:37:31.706 INFO:teuthology.orchestra.run.smithi136.stdout:Package: linux-image-generic 2023-11-16T01:37:31.706 INFO:teuthology.orchestra.run.smithi136.stdout:Status: install ok installed 2023-11-16T01:37:31.706 INFO:teuthology.orchestra.run.smithi136.stdout:Priority: optional 2023-11-16T01:37:31.706 INFO:teuthology.orchestra.run.smithi136.stdout:Section: kernel 2023-11-16T01:37:31.706 INFO:teuthology.orchestra.run.smithi136.stdout:Installed-Size: 20 2023-11-16T01:37:31.706 INFO:teuthology.orchestra.run.smithi136.stdout:Maintainer: Ubuntu Kernel Team 2023-11-16T01:37:31.707 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: amd64 2023-11-16T01:37:31.707 INFO:teuthology.orchestra.run.smithi136.stdout:Source: linux-meta 2023-11-16T01:37:31.707 INFO:teuthology.orchestra.run.smithi136.stdout:Version: 5.4.0.166.163 2023-11-16T01:37:31.707 INFO:teuthology.orchestra.run.smithi136.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-11-16T01:37:31.707 INFO:teuthology.orchestra.run.smithi136.stdout:Depends: linux-image-5.4.0-166-generic, linux-modules-extra-5.4.0-166-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-16T01:37:31.707 INFO:teuthology.orchestra.run.smithi136.stdout:Recommends: thermald 2023-11-16T01:37:31.707 INFO:teuthology.orchestra.run.smithi136.stdout:Description: Generic Linux kernel image 2023-11-16T01:37:31.707 INFO:teuthology.orchestra.run.smithi136.stdout: This package will always depend on the latest generic kernel image 2023-11-16T01:37:31.707 INFO:teuthology.orchestra.run.smithi136.stdout: available. 2023-11-16T01:37:31.707 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2023-11-16T01:37:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.h7glWNzUiH 2023-11-16T01:37:31.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo cp /boot/grub/grub.cfg /tmp/tmp.h7glWNzUiH 2023-11-16T01:37:31.772 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.h7glWNzUiH 2023-11-16T01:37:31.809 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.h7glWNzUiH is 9KB 2023-11-16T01:37:31.823 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.h7glWNzUiH 2023-11-16T01:37:31.830 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-16T01:37:31.888 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:37:31.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-16T01:37:31.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-16T01:37:31.930 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-166-generic 2023-11-16T01:37:31.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo update-grub 2023-11-16T01:37:33.000 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2023-11-16T01:37:33.025 DEBUG:teuthology.orchestra.run.smithi083:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-16T01:37:33.095 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2023-11-16T01:37:33.214 INFO:teuthology.orchestra.run.smithi136.stderr:Sourcing file `/etc/default/grub' 2023-11-16T01:37:33.226 INFO:teuthology.orchestra.run.smithi136.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-16T01:37:33.234 INFO:teuthology.orchestra.run.smithi136.stderr:Generating grub configuration file ... 2023-11-16T01:37:33.288 INFO:teuthology.orchestra.run.smithi083.stdout:Building dependency tree... 2023-11-16T01:37:33.289 INFO:teuthology.orchestra.run.smithi083.stdout:Reading state information... 2023-11-16T01:37:33.498 INFO:teuthology.orchestra.run.smithi083.stdout:linux-image-generic is already the newest version (5.4.0.166.163). 2023-11-16T01:37:33.499 INFO:teuthology.orchestra.run.smithi083.stdout:0 upgraded, 0 newly installed, 0 to remove and 278 not upgraded. 2023-11-16T01:37:33.500 DEBUG:teuthology.orchestra.run.smithi083:> dpkg -s linux-image-generic 2023-11-16T01:37:33.507 INFO:teuthology.orchestra.run.smithi136.stderr:Found linux image: /boot/vmlinuz-5.4.0-166-generic 2023-11-16T01:37:33.519 INFO:teuthology.orchestra.run.smithi136.stderr:Found initrd image: /boot/initrd.img-5.4.0-166-generic 2023-11-16T01:37:33.520 INFO:teuthology.orchestra.run.smithi083.stdout:Package: linux-image-generic 2023-11-16T01:37:33.520 INFO:teuthology.orchestra.run.smithi083.stdout:Status: install ok installed 2023-11-16T01:37:33.520 INFO:teuthology.orchestra.run.smithi083.stdout:Priority: optional 2023-11-16T01:37:33.520 INFO:teuthology.orchestra.run.smithi083.stdout:Section: kernel 2023-11-16T01:37:33.520 INFO:teuthology.orchestra.run.smithi083.stdout:Installed-Size: 20 2023-11-16T01:37:33.520 INFO:teuthology.orchestra.run.smithi083.stdout:Maintainer: Ubuntu Kernel Team 2023-11-16T01:37:33.520 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: amd64 2023-11-16T01:37:33.520 INFO:teuthology.orchestra.run.smithi083.stdout:Source: linux-meta 2023-11-16T01:37:33.520 INFO:teuthology.orchestra.run.smithi083.stdout:Version: 5.4.0.166.163 2023-11-16T01:37:33.521 INFO:teuthology.orchestra.run.smithi083.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-11-16T01:37:33.521 INFO:teuthology.orchestra.run.smithi083.stdout:Depends: linux-image-5.4.0-166-generic, linux-modules-extra-5.4.0-166-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-16T01:37:33.521 INFO:teuthology.orchestra.run.smithi083.stdout:Recommends: thermald 2023-11-16T01:37:33.521 INFO:teuthology.orchestra.run.smithi083.stdout:Description: Generic Linux kernel image 2023-11-16T01:37:33.521 INFO:teuthology.orchestra.run.smithi083.stdout: This package will always depend on the latest generic kernel image 2023-11-16T01:37:33.521 INFO:teuthology.orchestra.run.smithi083.stdout: available. 2023-11-16T01:37:33.521 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2023-11-16T01:37:33.526 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.Zo63MWq3Cb 2023-11-16T01:37:33.527 DEBUG:teuthology.orchestra.run.smithi083:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Zo63MWq3Cb 2023-11-16T01:37:33.536 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.Zo63MWq3Cb 2023-11-16T01:37:33.620 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.Zo63MWq3Cb is 9KB 2023-11-16T01:37:33.634 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.Zo63MWq3Cb 2023-11-16T01:37:33.641 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-16T01:37:33.697 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:37:33.697 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-16T01:37:33.697 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-16T01:37:33.727 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-166-generic 2023-11-16T01:37:33.727 DEBUG:teuthology.orchestra.run.smithi083:> sudo update-grub 2023-11-16T01:37:33.921 INFO:teuthology.orchestra.run.smithi136.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-11-16T01:37:33.933 INFO:teuthology.orchestra.run.smithi136.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-11-16T01:37:34.669 INFO:teuthology.orchestra.run.smithi136.stderr:done 2023-11-16T01:37:34.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo shutdown -r now 2023-11-16T01:37:35.024 INFO:teuthology.orchestra.run.smithi083.stderr:Sourcing file `/etc/default/grub' 2023-11-16T01:37:35.034 INFO:teuthology.orchestra.run.smithi083.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-16T01:37:35.047 INFO:teuthology.orchestra.run.smithi083.stderr:Generating grub configuration file ... 2023-11-16T01:37:35.330 INFO:teuthology.orchestra.run.smithi083.stderr:Found linux image: /boot/vmlinuz-5.4.0-166-generic 2023-11-16T01:37:35.342 INFO:teuthology.orchestra.run.smithi083.stderr:Found initrd image: /boot/initrd.img-5.4.0-166-generic 2023-11-16T01:37:35.707 INFO:teuthology.orchestra.run.smithi083.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-11-16T01:37:35.718 INFO:teuthology.orchestra.run.smithi083.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-11-16T01:37:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:done 2023-11-16T01:37:36.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo shutdown -r now 2023-11-16T01:38:04.699 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-11-16T01:38:04.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-11-16T01:38:04.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T01:38:06.457 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-16T01:38:06.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2023-11-16T01:38:06.458 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T01:38:23.226 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2023-11-16T01:38:25.018 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-11-16T01:38:32.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-11-16T01:38:32.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T01:38:34.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2023-11-16T01:38:34.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T01:38:35.294 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2023-11-16T01:38:37.082 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-11-16T01:38:47.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-11-16T01:38:47.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T01:38:49.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2023-11-16T01:38:49.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T01:38:52.154 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-11-16T01:39:07.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2023-11-16T01:39:07.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T01:39:10.238 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-11-16T01:39:18.978 DEBUG:teuthology.orchestra.run.smithi136:> true 2023-11-16T01:39:19.542 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-11-16T01:39:19.542 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-166-generic"... 2023-11-16T01:39:19.542 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2023-11-16T01:39:19.549 INFO:teuthology.orchestra.run.smithi136.stdout:5.4.0-166-generic 2023-11-16T01:39:19.549 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-166-generic vs 5.4.0-166-generic 2023-11-16T01:39:19.549 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-16T01:39:19.550 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-16T01:39:20.551 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-11-16T01:39:20.551 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-16T01:39:20.640 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2023-11-16T01:39:20.666 DEBUG:teuthology.parallel:result is None 2023-11-16T01:39:28.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2023-11-16T01:39:28.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T01:39:28.444 DEBUG:teuthology.orchestra.run.smithi083:> true 2023-11-16T01:39:29.011 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2023-11-16T01:39:29.011 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-166-generic"... 2023-11-16T01:39:29.011 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2023-11-16T01:39:29.018 INFO:teuthology.orchestra.run.smithi083.stdout:5.4.0-166-generic 2023-11-16T01:39:29.019 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-166-generic vs 5.4.0-166-generic 2023-11-16T01:39:29.019 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-16T01:39:29.019 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-16T01:39:30.019 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-16T01:39:30.020 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-16T01:39:30.109 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1 2023-11-16T01:39:30.140 DEBUG:teuthology.parallel:result is None 2023-11-16T01:39:30.140 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-16T01:39:30.147 INFO:teuthology.task.internal:Creating test directory... 2023-11-16T01:39:30.147 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T01:39:30.149 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T01:39:30.157 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-16T01:39:30.191 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-16T01:39:30.236 INFO:teuthology.task.internal:Creating archive directory... 2023-11-16T01:39:30.236 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T01:39:30.238 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T01:39:30.265 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-16T01:39:30.286 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-16T01:39:30.286 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T01:39:30.296 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T01:39:30.311 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T01:39:30.317 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T01:39:30.322 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T01:39:30.326 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T01:39:30.328 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-16T01:39:30.336 INFO:teuthology.task.internal:Configuring sudo... 2023-11-16T01:39:30.336 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T01:39:30.338 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T01:39:30.387 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-16T01:39:30.396 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-16T01:39:30.396 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T01:39:30.398 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T01:39:30.404 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T01:39:30.412 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T01:39:30.417 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:39:30.418 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T01:39:30.436 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T01:39:30.443 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T01:39:30.495 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:39:30.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T01:39:30.518 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2023-11-16T01:39:30.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2023-11-16T01:39:30.568 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-16T01:39:30.576 INFO:teuthology.task.internal:Starting timer... 2023-11-16T01:39:30.576 INFO:teuthology.run_tasks:Running task pcp... 2023-11-16T01:39:30.645 INFO:teuthology.run_tasks:Running task selinux... 2023-11-16T01:39:30.699 DEBUG:teuthology.task.selinux:Excluding smithi083: OS 'ubuntu' does not support SELinux 2023-11-16T01:39:30.699 DEBUG:teuthology.task.selinux:Excluding smithi136: OS 'ubuntu' does not support SELinux 2023-11-16T01:39:30.699 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-16T01:39:30.699 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-11-16T01:39:30.699 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-16T01:39:30.699 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-16T01:39:30.748 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-16T01:39:30.748 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-16T01:39:30.769 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-16T01:39:30.771 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi083.front.sepia.ceph.com,smithi136.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-16T01:45:31.056 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi083.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com')] 2023-11-16T01:45:31.057 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2023-11-16T01:45:31.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T01:45:31.110 DEBUG:teuthology.orchestra.run.smithi083:> true 2023-11-16T01:45:31.233 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2023-11-16T01:45:31.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-11-16T01:45:31.234 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T01:45:31.280 DEBUG:teuthology.orchestra.run.smithi136:> true 2023-11-16T01:45:31.404 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-11-16T01:45:31.404 INFO:teuthology.run_tasks:Running task clock... 2023-11-16T01:45:31.415 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-16T01:45:31.415 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T01:45:31.416 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T01:45:31.418 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T01:45:31.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T01:45:31.451 INFO:teuthology.orchestra.run.smithi083.stdout:16 Nov 01:45:31 ntpd[23621]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-11-16T01:45:31.452 INFO:teuthology.orchestra.run.smithi083.stdout:16 Nov 01:45:31 ntpd[23621]: Command line: ntpd -gq 2023-11-16T01:45:31.452 INFO:teuthology.orchestra.run.smithi083.stdout:16 Nov 01:45:31 ntpd[23621]: proto: precision = 0.060 usec (-24) 2023-11-16T01:45:31.452 INFO:teuthology.orchestra.run.smithi083.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-16T01:45:31.453 INFO:teuthology.orchestra.run.smithi083.stdout:16 Nov 01:45:31 ntpd[23621]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-16T01:45:31.453 INFO:teuthology.orchestra.run.smithi083.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-16T01:45:31.454 INFO:teuthology.orchestra.run.smithi083.stdout:16 Nov 01:45:31 ntpd[23621]: restrict ::: KOD does nothing without LIMITED. 2023-11-16T01:45:31.454 INFO:teuthology.orchestra.run.smithi083.stdout:16 Nov 01:45:31 ntpd[23621]: Listen and drop on 0 v6wildcard [::]:123 2023-11-16T01:45:31.454 INFO:teuthology.orchestra.run.smithi083.stdout:16 Nov 01:45:31 ntpd[23621]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-16T01:45:31.454 INFO:teuthology.orchestra.run.smithi083.stdout:16 Nov 01:45:31 ntpd[23621]: Listen normally on 2 lo 127.0.0.1:123 2023-11-16T01:45:31.455 INFO:teuthology.orchestra.run.smithi083.stdout:16 Nov 01:45:31 ntpd[23621]: Listen normally on 3 enp3s0f1 172.21.15.83:123 2023-11-16T01:45:31.455 INFO:teuthology.orchestra.run.smithi083.stdout:16 Nov 01:45:31 ntpd[23621]: Listen normally on 4 lo [::1]:123 2023-11-16T01:45:31.455 INFO:teuthology.orchestra.run.smithi083.stdout:16 Nov 01:45:31 ntpd[23621]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce91%5]:123 2023-11-16T01:45:31.455 INFO:teuthology.orchestra.run.smithi083.stdout:16 Nov 01:45:31 ntpd[23621]: Listening on routing socket on fd #22 for interface updates 2023-11-16T01:45:31.455 INFO:teuthology.orchestra.run.smithi136.stdout:16 Nov 01:45:31 ntpd[23683]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-11-16T01:45:31.456 INFO:teuthology.orchestra.run.smithi136.stdout:16 Nov 01:45:31 ntpd[23683]: Command line: ntpd -gq 2023-11-16T01:45:31.456 INFO:teuthology.orchestra.run.smithi136.stdout:16 Nov 01:45:31 ntpd[23683]: proto: precision = 0.100 usec (-23) 2023-11-16T01:45:31.456 INFO:teuthology.orchestra.run.smithi136.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-16T01:45:31.456 INFO:teuthology.orchestra.run.smithi136.stdout:16 Nov 01:45:31 ntpd[23683]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-16T01:45:31.457 INFO:teuthology.orchestra.run.smithi136.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-16T01:45:31.458 INFO:teuthology.orchestra.run.smithi136.stdout:16 Nov 01:45:31 ntpd[23683]: restrict ::: KOD does nothing without LIMITED. 2023-11-16T01:45:31.458 INFO:teuthology.orchestra.run.smithi136.stdout:16 Nov 01:45:31 ntpd[23683]: Listen and drop on 0 v6wildcard [::]:123 2023-11-16T01:45:31.458 INFO:teuthology.orchestra.run.smithi136.stdout:16 Nov 01:45:31 ntpd[23683]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-16T01:45:31.458 INFO:teuthology.orchestra.run.smithi136.stdout:16 Nov 01:45:31 ntpd[23683]: Listen normally on 2 lo 127.0.0.1:123 2023-11-16T01:45:31.458 INFO:teuthology.orchestra.run.smithi136.stdout:16 Nov 01:45:31 ntpd[23683]: Listen normally on 3 enp3s0f1 172.21.15.136:123 2023-11-16T01:45:31.458 INFO:teuthology.orchestra.run.smithi136.stdout:16 Nov 01:45:31 ntpd[23683]: Listen normally on 4 lo [::1]:123 2023-11-16T01:45:31.458 INFO:teuthology.orchestra.run.smithi136.stdout:16 Nov 01:45:31 ntpd[23683]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e43%5]:123 2023-11-16T01:45:31.458 INFO:teuthology.orchestra.run.smithi136.stdout:16 Nov 01:45:31 ntpd[23683]: Listening on routing socket on fd #22 for interface updates 2023-11-16T01:45:32.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:32 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:32.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:32 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:32.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:32 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:32.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:32 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:32.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:32 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:32.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:32 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:32.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:32 ntpd[23621]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:32.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:32 ntpd[23621]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:32.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:32 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:32.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:32 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:32.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:32 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:32.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:32 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:32.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:32 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:32.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:32 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:32.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:32 ntpd[23683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:32.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:32 ntpd[23683]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:32.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:32 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:32.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:32 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:32.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:32 ntpd[23683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:32.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:32 ntpd[23683]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:33.452 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:33 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:33.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:33 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:33.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:33 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:33.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:33 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:33.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:33 ntpd[23621]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:33.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:33 ntpd[23621]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:34.452 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:34 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:34.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:34 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:34.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:34 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:34.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:34 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:34.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:34 ntpd[23621]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:34.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:34 ntpd[23621]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:34.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:34.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:34.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:34.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:34.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:34.455 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:34.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:34.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:34.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:34.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:34.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:34.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:34.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:34.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:34.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:34.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:34.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:34.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:34.457 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:34.457 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:34 ntpd[23683]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:35.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:35 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:35.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:35 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:35.454 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:35 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:35.454 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:35 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:35.454 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:35 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:35.454 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:35 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:35.454 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:35 ntpd[23621]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:35.455 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:35 ntpd[23621]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:35.455 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:35 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:35.455 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:35 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:35.455 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:35 ntpd[23621]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:35.455 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:35 ntpd[23621]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:36 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:36 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:36 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:36 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:36 ntpd[23621]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:36 ntpd[23621]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:36.457 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:36.457 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:36.457 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:36.457 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:36.457 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:36.457 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:36.457 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:36.458 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:36.458 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:36.458 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:36.458 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:36.458 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:36.458 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:36.458 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:36.458 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:36.458 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:36.458 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:36.458 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:36 ntpd[23683]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:37.454 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:37 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:37.454 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:37 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:37.454 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:37 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:37.454 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:37 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:37.455 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:37 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:37.455 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:37 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:37.455 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:37 ntpd[23621]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:37.455 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:37 ntpd[23621]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:37.455 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:37 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:37.456 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:37 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:37.456 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:37 ntpd[23621]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:37.456 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:37 ntpd[23621]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:38.452 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:38 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:38 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:38 ntpd[23621]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:38 ntpd[23621]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:38 ntpd[23621]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:38 ntpd[23621]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:38.453 INFO:teuthology.orchestra.run.smithi083.stdout:16 Nov 01:45:38 ntpd[23621]: ntpd: time slew -0.001051 s 2023-11-16T01:45:38.454 INFO:teuthology.orchestra.run.smithi083.stdout:ntpd: time slew -0.001051s 2023-11-16T01:45:38.454 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:38 ntpd[23621]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-16T01:45:38.454 INFO:teuthology.orchestra.run.smithi083.stderr:16 Nov 01:45:38 ntpd[23621]: can't open /var/log/ntpstats/loopstats.20231116: Permission denied 2023-11-16T01:45:38.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:38 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:38.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:38 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:38.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:38 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:38.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:38 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:38.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:38 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:38.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:38 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:38.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:38 ntpd[23683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-16T01:45:38.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:38 ntpd[23683]: can't open /var/log/ntpstats/rawstats.20231116: Permission denied 2023-11-16T01:45:38.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:38 ntpd[23683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-16T01:45:38.456 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:38 ntpd[23683]: can't open /var/log/ntpstats/peerstats.20231116: Permission denied 2023-11-16T01:45:38.457 INFO:teuthology.orchestra.run.smithi136.stdout:16 Nov 01:45:38 ntpd[23683]: ntpd: time slew +0.000850 s 2023-11-16T01:45:38.457 INFO:teuthology.orchestra.run.smithi136.stdout:ntpd: time slew +0.000850s 2023-11-16T01:45:38.457 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:38 ntpd[23683]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-16T01:45:38.457 INFO:teuthology.orchestra.run.smithi136.stderr:16 Nov 01:45:38 ntpd[23683]: can't open /var/log/ntpstats/loopstats.20231116: Permission denied 2023-11-16T01:45:38.519 INFO:teuthology.orchestra.run.smithi083.stdout: remote refid st t when poll reach delay offset jitter 2023-11-16T01:45:38.519 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================== 2023-11-16T01:45:38.519 INFO:teuthology.orchestra.run.smithi083.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-16T01:45:38.519 INFO:teuthology.orchestra.run.smithi083.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-16T01:45:38.520 INFO:teuthology.orchestra.run.smithi083.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-16T01:45:38.520 INFO:teuthology.orchestra.run.smithi083.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-16T01:45:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: remote refid st t when poll reach delay offset jitter 2023-11-16T01:45:38.521 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================== 2023-11-16T01:45:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-16T01:45:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-16T01:45:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-16T01:45:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-16T01:45:38.522 INFO:teuthology.run_tasks:Running task install... 2023-11-16T01:45:38.530 DEBUG:teuthology.task.install:project ceph 2023-11-16T01:45:38.530 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac'}} 2023-11-16T01:45:38.530 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac'} 2023-11-16T01:45:38.530 INFO:teuthology.task.install:Using flavor: default 2023-11-16T01:45:38.536 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-16T01:45:38.536 INFO:teuthology.task.install:extra packages: [] 2023-11-16T01:45:38.536 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-key list | grep Ceph 2023-11-16T01:45:38.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-key list | grep Ceph 2023-11-16T01:45:38.665 INFO:teuthology.orchestra.run.smithi136.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-11-16T01:45:38.669 INFO:teuthology.orchestra.run.smithi083.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-11-16T01:45:38.722 INFO:teuthology.orchestra.run.smithi136.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-16T01:45:38.722 INFO:teuthology.orchestra.run.smithi136.stdout:uid [ unknown] Ceph.com (release key) 2023-11-16T01:45:38.723 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-11-16T01:45:38.723 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-16T01:45:38.731 INFO:teuthology.orchestra.run.smithi083.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-16T01:45:38.731 INFO:teuthology.orchestra.run.smithi083.stdout:uid [ unknown] Ceph.com (release key) 2023-11-16T01:45:38.731 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-11-16T01:45:38.731 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-16T01:45:38.869 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default/ 2023-11-16T01:45:38.869 INFO:teuthology.task.install.deb:Package version is 16.2.14-171-ged3c8e92-1focal 2023-11-16T01:45:39.012 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default/ 2023-11-16T01:45:39.012 INFO:teuthology.task.install.deb:Package version is 16.2.14-171-ged3c8e92-1focal 2023-11-16T01:45:39.121 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:45:39.122 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-16T01:45:39.129 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:45:39.129 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-16T01:45:39.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2023-11-16T01:45:39.147 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-get update 2023-11-16T01:45:39.302 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-16T01:45:39.308 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-11-16T01:45:39.316 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-11-16T01:45:39.325 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-11-16T01:45:39.336 INFO:teuthology.orchestra.run.smithi083.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal InRelease 2023-11-16T01:45:39.338 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-16T01:45:39.338 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-11-16T01:45:39.347 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-11-16T01:45:39.358 INFO:teuthology.orchestra.run.smithi083.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal Release [7,680 B] 2023-11-16T01:45:39.367 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-11-16T01:45:39.381 INFO:teuthology.orchestra.run.smithi083.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal Release.gpg 2023-11-16T01:45:39.403 INFO:teuthology.orchestra.run.smithi083.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 Packages [17.2 kB] 2023-11-16T01:45:39.434 INFO:teuthology.orchestra.run.smithi083.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main i386 Packages [3,111 B] 2023-11-16T01:45:39.444 INFO:teuthology.orchestra.run.smithi136.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal InRelease 2023-11-16T01:45:39.502 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal Release [7,680 B] 2023-11-16T01:45:39.540 INFO:teuthology.orchestra.run.smithi136.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal Release.gpg 2023-11-16T01:45:39.584 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 Packages [17.2 kB] 2023-11-16T01:45:39.694 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main i386 Packages [3,111 B] 2023-11-16T01:45:39.968 INFO:teuthology.orchestra.run.smithi083.stdout:Fetched 28.0 kB in 1s (42.6 kB/s) 2023-11-16T01:45:39.980 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 28.0 kB in 1s (44.0 kB/s) 2023-11-16T01:45:41.011 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2023-11-16T01:45:41.037 DEBUG:teuthology.orchestra.run.smithi083:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.2.14-171-ged3c8e92-1focal cephadm=16.2.14-171-ged3c8e92-1focal ceph-mds=16.2.14-171-ged3c8e92-1focal ceph-mgr=16.2.14-171-ged3c8e92-1focal ceph-common=16.2.14-171-ged3c8e92-1focal ceph-fuse=16.2.14-171-ged3c8e92-1focal ceph-test=16.2.14-171-ged3c8e92-1focal radosgw=16.2.14-171-ged3c8e92-1focal python3-rados=16.2.14-171-ged3c8e92-1focal python3-rgw=16.2.14-171-ged3c8e92-1focal python3-cephfs=16.2.14-171-ged3c8e92-1focal python3-rbd=16.2.14-171-ged3c8e92-1focal libcephfs2=16.2.14-171-ged3c8e92-1focal libcephfs-dev=16.2.14-171-ged3c8e92-1focal librados2=16.2.14-171-ged3c8e92-1focal librbd1=16.2.14-171-ged3c8e92-1focal rbd-fuse=16.2.14-171-ged3c8e92-1focal 2023-11-16T01:45:41.040 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2023-11-16T01:45:41.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.2.14-171-ged3c8e92-1focal cephadm=16.2.14-171-ged3c8e92-1focal ceph-mds=16.2.14-171-ged3c8e92-1focal ceph-mgr=16.2.14-171-ged3c8e92-1focal ceph-common=16.2.14-171-ged3c8e92-1focal ceph-fuse=16.2.14-171-ged3c8e92-1focal ceph-test=16.2.14-171-ged3c8e92-1focal radosgw=16.2.14-171-ged3c8e92-1focal python3-rados=16.2.14-171-ged3c8e92-1focal python3-rgw=16.2.14-171-ged3c8e92-1focal python3-cephfs=16.2.14-171-ged3c8e92-1focal python3-rbd=16.2.14-171-ged3c8e92-1focal libcephfs2=16.2.14-171-ged3c8e92-1focal libcephfs-dev=16.2.14-171-ged3c8e92-1focal librados2=16.2.14-171-ged3c8e92-1focal librbd1=16.2.14-171-ged3c8e92-1focal rbd-fuse=16.2.14-171-ged3c8e92-1focal 2023-11-16T01:45:41.106 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2023-11-16T01:45:41.140 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2023-11-16T01:45:41.284 INFO:teuthology.orchestra.run.smithi083.stdout:Building dependency tree... 2023-11-16T01:45:41.285 INFO:teuthology.orchestra.run.smithi083.stdout:Reading state information... 2023-11-16T01:45:41.341 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2023-11-16T01:45:41.342 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2023-11-16T01:45:41.453 INFO:teuthology.orchestra.run.smithi083.stdout:The following packages were automatically installed and are no longer required: 2023-11-16T01:45:41.454 INFO:teuthology.orchestra.run.smithi083.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-11-16T01:45:41.454 INFO:teuthology.orchestra.run.smithi083.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-16T01:45:41.454 INFO:teuthology.orchestra.run.smithi083.stdout:The following additional packages will be installed: 2023-11-16T01:45:41.454 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-16T01:45:41.455 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq libjaeger 2023-11-16T01:45:41.455 INFO:teuthology.orchestra.run.smithi083.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2023-11-16T01:45:41.455 INFO:teuthology.orchestra.run.smithi083.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 2023-11-16T01:45:41.455 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph nvme-cli python-pastedeploy-tpl 2023-11-16T01:45:41.455 INFO:teuthology.orchestra.run.smithi083.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2023-11-16T01:45:41.455 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2023-11-16T01:45:41.455 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2023-11-16T01:45:41.455 INFO:teuthology.orchestra.run.smithi083.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-11-16T01:45:41.455 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-11-16T01:45:41.455 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2023-11-16T01:45:41.456 INFO:teuthology.orchestra.run.smithi083.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-16T01:45:41.456 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-11-16T01:45:41.456 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-11-16T01:45:41.456 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-11-16T01:45:41.456 INFO:teuthology.orchestra.run.smithi083.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-16T01:45:41.456 INFO:teuthology.orchestra.run.smithi083.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-11-16T01:45:41.456 INFO:teuthology.orchestra.run.smithi083.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-11-16T01:45:41.456 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2023-11-16T01:45:41.456 INFO:teuthology.orchestra.run.smithi083.stdout:Suggested packages: 2023-11-16T01:45:41.457 INFO:teuthology.orchestra.run.smithi083.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2023-11-16T01:45:41.457 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-11-16T01:45:41.457 INFO:teuthology.orchestra.run.smithi083.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-11-16T01:45:41.457 INFO:teuthology.orchestra.run.smithi083.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-11-16T01:45:41.457 INFO:teuthology.orchestra.run.smithi083.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-11-16T01:45:41.457 INFO:teuthology.orchestra.run.smithi083.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-11-16T01:45:41.457 INFO:teuthology.orchestra.run.smithi083.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-11-16T01:45:41.457 INFO:teuthology.orchestra.run.smithi083.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-11-16T01:45:41.457 INFO:teuthology.orchestra.run.smithi083.stdout:Recommended packages: 2023-11-16T01:45:41.457 INFO:teuthology.orchestra.run.smithi083.stdout: btrfs-tools 2023-11-16T01:45:41.521 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages were automatically installed and are no longer required: 2023-11-16T01:45:41.522 INFO:teuthology.orchestra.run.smithi136.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-11-16T01:45:41.523 INFO:teuthology.orchestra.run.smithi136.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-16T01:45:41.523 INFO:teuthology.orchestra.run.smithi136.stdout:The following additional packages will be installed: 2023-11-16T01:45:41.523 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-16T01:45:41.524 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq libjaeger 2023-11-16T01:45:41.524 INFO:teuthology.orchestra.run.smithi136.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2023-11-16T01:45:41.524 INFO:teuthology.orchestra.run.smithi136.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 2023-11-16T01:45:41.524 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph nvme-cli python-pastedeploy-tpl 2023-11-16T01:45:41.524 INFO:teuthology.orchestra.run.smithi136.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2023-11-16T01:45:41.524 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2023-11-16T01:45:41.525 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2023-11-16T01:45:41.525 INFO:teuthology.orchestra.run.smithi136.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-11-16T01:45:41.525 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-11-16T01:45:41.525 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2023-11-16T01:45:41.525 INFO:teuthology.orchestra.run.smithi136.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-16T01:45:41.525 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-11-16T01:45:41.525 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-11-16T01:45:41.525 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-11-16T01:45:41.525 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-16T01:45:41.525 INFO:teuthology.orchestra.run.smithi136.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-11-16T01:45:41.525 INFO:teuthology.orchestra.run.smithi136.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-11-16T01:45:41.525 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2023-11-16T01:45:41.526 INFO:teuthology.orchestra.run.smithi136.stdout:Suggested packages: 2023-11-16T01:45:41.526 INFO:teuthology.orchestra.run.smithi136.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2023-11-16T01:45:41.526 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-11-16T01:45:41.526 INFO:teuthology.orchestra.run.smithi136.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-11-16T01:45:41.526 INFO:teuthology.orchestra.run.smithi136.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-11-16T01:45:41.526 INFO:teuthology.orchestra.run.smithi136.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-11-16T01:45:41.527 INFO:teuthology.orchestra.run.smithi136.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-11-16T01:45:41.527 INFO:teuthology.orchestra.run.smithi136.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-11-16T01:45:41.527 INFO:teuthology.orchestra.run.smithi136.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-11-16T01:45:41.527 INFO:teuthology.orchestra.run.smithi136.stdout:Recommended packages: 2023-11-16T01:45:41.527 INFO:teuthology.orchestra.run.smithi136.stdout: btrfs-tools 2023-11-16T01:45:41.564 INFO:teuthology.orchestra.run.smithi083.stdout:The following NEW packages will be installed: 2023-11-16T01:45:41.564 INFO:teuthology.orchestra.run.smithi083.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-16T01:45:41.564 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-16T01:45:41.564 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2023-11-16T01:45:41.564 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2 libjaeger libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2023-11-16T01:45:41.564 INFO:teuthology.orchestra.run.smithi083.stdout: libleveldb1d liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 2023-11-16T01:45:41.565 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph nvme-cli 2023-11-16T01:45:41.565 INFO:teuthology.orchestra.run.smithi083.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-11-16T01:45:41.565 INFO:teuthology.orchestra.run.smithi083.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-11-16T01:45:41.565 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-11-16T01:45:41.565 INFO:teuthology.orchestra.run.smithi083.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-11-16T01:45:41.565 INFO:teuthology.orchestra.run.smithi083.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-11-16T01:45:41.565 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2023-11-16T01:45:41.566 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-11-16T01:45:41.566 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-11-16T01:45:41.566 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-11-16T01:45:41.566 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2023-11-16T01:45:41.566 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2023-11-16T01:45:41.566 INFO:teuthology.orchestra.run.smithi083.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-11-16T01:45:41.566 INFO:teuthology.orchestra.run.smithi083.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2023-11-16T01:45:41.566 INFO:teuthology.orchestra.run.smithi083.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2023-11-16T01:45:41.566 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2023-11-16T01:45:41.567 INFO:teuthology.orchestra.run.smithi083.stdout:The following packages will be upgraded: 2023-11-16T01:45:41.567 INFO:teuthology.orchestra.run.smithi083.stdout: librados2 librbd1 2023-11-16T01:45:41.633 INFO:teuthology.orchestra.run.smithi136.stdout:The following NEW packages will be installed: 2023-11-16T01:45:41.633 INFO:teuthology.orchestra.run.smithi136.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-16T01:45:41.633 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-16T01:45:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2023-11-16T01:45:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 libjaeger libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2023-11-16T01:45:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: libleveldb1d liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 2023-11-16T01:45:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph nvme-cli 2023-11-16T01:45:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-11-16T01:45:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-11-16T01:45:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-11-16T01:45:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-11-16T01:45:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-11-16T01:45:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2023-11-16T01:45:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-11-16T01:45:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-11-16T01:45:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-11-16T01:45:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2023-11-16T01:45:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2023-11-16T01:45:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-11-16T01:45:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2023-11-16T01:45:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2023-11-16T01:45:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2023-11-16T01:45:41.637 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages will be upgraded: 2023-11-16T01:45:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 librbd1 2023-11-16T01:45:41.695 INFO:teuthology.orchestra.run.smithi083.stdout:2 upgraded, 98 newly installed, 0 to remove and 257 not upgraded. 2023-11-16T01:45:41.695 INFO:teuthology.orchestra.run.smithi083.stdout:Need to get 174 MB of archives. 2023-11-16T01:45:41.695 INFO:teuthology.orchestra.run.smithi083.stdout:After this operation, 805 MB of additional disk space will be used. 2023-11-16T01:45:41.695 INFO:teuthology.orchestra.run.smithi083.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 libjaeger amd64 16.2.14-171-ged3c8e92-1focal [3,968 B] 2023-11-16T01:45:41.702 INFO:teuthology.orchestra.run.smithi136.stdout:2 upgraded, 98 newly installed, 0 to remove and 257 not upgraded. 2023-11-16T01:45:41.702 INFO:teuthology.orchestra.run.smithi136.stdout:Need to get 174 MB of archives. 2023-11-16T01:45:41.702 INFO:teuthology.orchestra.run.smithi136.stdout:After this operation, 805 MB of additional disk space will be used. 2023-11-16T01:45:41.702 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-11-16T01:45:41.737 INFO:teuthology.orchestra.run.smithi083.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 16.2.14-171-ged3c8e92-1focal [3,377 kB] 2023-11-16T01:45:41.758 INFO:teuthology.orchestra.run.smithi083.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-11-16T01:45:41.759 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 libjaeger amd64 16.2.14-171-ged3c8e92-1focal [3,968 B] 2023-11-16T01:45:41.801 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-11-16T01:45:41.801 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-11-16T01:45:41.802 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 16.2.14-171-ged3c8e92-1focal [3,377 kB] 2023-11-16T01:45:41.809 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-11-16T01:45:41.821 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-11-16T01:45:41.850 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2023-11-16T01:45:41.859 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-11-16T01:45:41.861 INFO:teuthology.orchestra.run.smithi136.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-11-16T01:45:41.869 INFO:teuthology.orchestra.run.smithi136.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-11-16T01:45:41.870 INFO:teuthology.orchestra.run.smithi136.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-11-16T01:45:41.871 INFO:teuthology.orchestra.run.smithi136.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-11-16T01:45:41.875 INFO:teuthology.orchestra.run.smithi136.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-11-16T01:45:41.875 INFO:teuthology.orchestra.run.smithi136.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-11-16T01:45:41.876 INFO:teuthology.orchestra.run.smithi136.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-11-16T01:45:41.886 INFO:teuthology.orchestra.run.smithi136.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-11-16T01:45:41.888 INFO:teuthology.orchestra.run.smithi136.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-16T01:45:41.888 INFO:teuthology.orchestra.run.smithi136.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-11-16T01:45:41.907 INFO:teuthology.orchestra.run.smithi136.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-11-16T01:45:41.908 INFO:teuthology.orchestra.run.smithi136.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-11-16T01:45:41.909 INFO:teuthology.orchestra.run.smithi136.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-11-16T01:45:41.911 INFO:teuthology.orchestra.run.smithi136.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-11-16T01:45:41.913 INFO:teuthology.orchestra.run.smithi136.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-11-16T01:45:41.914 INFO:teuthology.orchestra.run.smithi136.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-11-16T01:45:41.920 INFO:teuthology.orchestra.run.smithi136.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-11-16T01:45:41.929 INFO:teuthology.orchestra.run.smithi136.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-11-16T01:45:41.929 INFO:teuthology.orchestra.run.smithi136.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-11-16T01:45:41.930 INFO:teuthology.orchestra.run.smithi136.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-11-16T01:45:41.949 INFO:teuthology.orchestra.run.smithi136.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-11-16T01:45:41.953 INFO:teuthology.orchestra.run.smithi136.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-11-16T01:45:41.954 INFO:teuthology.orchestra.run.smithi136.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-16T01:45:41.955 INFO:teuthology.orchestra.run.smithi136.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-11-16T01:45:41.957 INFO:teuthology.orchestra.run.smithi136.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-11-16T01:45:41.958 INFO:teuthology.orchestra.run.smithi136.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-11-16T01:45:41.959 INFO:teuthology.orchestra.run.smithi136.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-11-16T01:45:41.971 INFO:teuthology.orchestra.run.smithi136.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-11-16T01:45:41.972 INFO:teuthology.orchestra.run.smithi136.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-11-16T01:45:42.145 INFO:teuthology.orchestra.run.smithi136.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-11-16T01:45:42.155 INFO:teuthology.orchestra.run.smithi083.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-11-16T01:45:42.162 INFO:teuthology.orchestra.run.smithi083.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-11-16T01:45:42.182 INFO:teuthology.orchestra.run.smithi136.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-11-16T01:45:42.186 INFO:teuthology.orchestra.run.smithi136.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-11-16T01:45:42.207 INFO:teuthology.orchestra.run.smithi083.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-11-16T01:45:42.211 INFO:teuthology.orchestra.run.smithi136.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-11-16T01:45:42.211 INFO:teuthology.orchestra.run.smithi136.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-11-16T01:45:42.211 INFO:teuthology.orchestra.run.smithi136.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-11-16T01:45:42.212 INFO:teuthology.orchestra.run.smithi136.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-11-16T01:45:42.212 INFO:teuthology.orchestra.run.smithi136.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-11-16T01:45:42.213 INFO:teuthology.orchestra.run.smithi136.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-11-16T01:45:42.213 INFO:teuthology.orchestra.run.smithi136.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-11-16T01:45:42.221 INFO:teuthology.orchestra.run.smithi083.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-11-16T01:45:42.228 INFO:teuthology.orchestra.run.smithi136.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-11-16T01:45:42.228 INFO:teuthology.orchestra.run.smithi136.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-11-16T01:45:42.247 INFO:teuthology.orchestra.run.smithi136.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-11-16T01:45:42.255 INFO:teuthology.orchestra.run.smithi136.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-11-16T01:45:42.258 INFO:teuthology.orchestra.run.smithi136.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-11-16T01:45:42.259 INFO:teuthology.orchestra.run.smithi136.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-11-16T01:45:42.260 INFO:teuthology.orchestra.run.smithi136.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-11-16T01:45:42.263 INFO:teuthology.orchestra.run.smithi136.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-11-16T01:45:42.266 INFO:teuthology.orchestra.run.smithi136.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-11-16T01:45:42.268 INFO:teuthology.orchestra.run.smithi136.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-11-16T01:45:42.271 INFO:teuthology.orchestra.run.smithi136.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-11-16T01:45:42.271 INFO:teuthology.orchestra.run.smithi083.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 16.2.14-171-ged3c8e92-1focal [3,394 kB] 2023-11-16T01:45:42.272 INFO:teuthology.orchestra.run.smithi136.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-11-16T01:45:42.291 INFO:teuthology.orchestra.run.smithi136.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-11-16T01:45:42.292 INFO:teuthology.orchestra.run.smithi136.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-11-16T01:45:42.308 INFO:teuthology.orchestra.run.smithi136.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-11-16T01:45:42.308 INFO:teuthology.orchestra.run.smithi136.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-16T01:45:42.308 INFO:teuthology.orchestra.run.smithi136.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-11-16T01:45:42.309 INFO:teuthology.orchestra.run.smithi136.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-11-16T01:45:42.310 INFO:teuthology.orchestra.run.smithi136.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-11-16T01:45:42.311 INFO:teuthology.orchestra.run.smithi136.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-11-16T01:45:42.319 INFO:teuthology.orchestra.run.smithi136.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-11-16T01:45:42.320 INFO:teuthology.orchestra.run.smithi136.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-11-16T01:45:42.340 INFO:teuthology.orchestra.run.smithi136.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-11-16T01:45:42.340 INFO:teuthology.orchestra.run.smithi136.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 16.2.14-171-ged3c8e92-1focal [3,394 kB] 2023-11-16T01:45:42.347 INFO:teuthology.orchestra.run.smithi083.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2023-11-16T01:45:42.370 INFO:teuthology.orchestra.run.smithi083.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-11-16T01:45:42.380 INFO:teuthology.orchestra.run.smithi083.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-11-16T01:45:42.407 INFO:teuthology.orchestra.run.smithi083.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-11-16T01:45:42.412 INFO:teuthology.orchestra.run.smithi083.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-11-16T01:45:42.415 INFO:teuthology.orchestra.run.smithi083.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-11-16T01:45:42.502 INFO:teuthology.orchestra.run.smithi083.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 16.2.14-171-ged3c8e92-1focal [729 kB] 2023-11-16T01:45:42.546 INFO:teuthology.orchestra.run.smithi083.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 16.2.14-171-ged3c8e92-1focal [413 kB] 2023-11-16T01:45:42.571 INFO:teuthology.orchestra.run.smithi083.stdout:Get:17 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 16.2.14-171-ged3c8e92-1focal [22.2 kB] 2023-11-16T01:45:42.572 INFO:teuthology.orchestra.run.smithi083.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 16.2.14-171-ged3c8e92-1focal [212 kB] 2023-11-16T01:45:42.581 INFO:teuthology.orchestra.run.smithi083.stdout:Get:19 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 16.2.14-171-ged3c8e92-1focal [38.5 kB] 2023-11-16T01:45:42.582 INFO:teuthology.orchestra.run.smithi083.stdout:Get:20 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 16.2.14-171-ged3c8e92-1focal [385 kB] 2023-11-16T01:45:42.587 INFO:teuthology.orchestra.run.smithi136.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 16.2.14-171-ged3c8e92-1focal [729 kB] 2023-11-16T01:45:42.594 INFO:teuthology.orchestra.run.smithi083.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-11-16T01:45:42.608 INFO:teuthology.orchestra.run.smithi083.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 16.2.14-171-ged3c8e92-1focal [3,703 kB] 2023-11-16T01:45:42.636 INFO:teuthology.orchestra.run.smithi136.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 16.2.14-171-ged3c8e92-1focal [413 kB] 2023-11-16T01:45:42.665 INFO:teuthology.orchestra.run.smithi136.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 16.2.14-171-ged3c8e92-1focal [22.2 kB] 2023-11-16T01:45:42.672 INFO:teuthology.orchestra.run.smithi136.stdout:Get:76 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 16.2.14-171-ged3c8e92-1focal [212 kB] 2023-11-16T01:45:42.679 INFO:teuthology.orchestra.run.smithi136.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 16.2.14-171-ged3c8e92-1focal [38.5 kB] 2023-11-16T01:45:42.680 INFO:teuthology.orchestra.run.smithi136.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 16.2.14-171-ged3c8e92-1focal [385 kB] 2023-11-16T01:45:42.712 INFO:teuthology.orchestra.run.smithi136.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 16.2.14-171-ged3c8e92-1focal [3,703 kB] 2023-11-16T01:45:42.762 INFO:teuthology.orchestra.run.smithi083.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-11-16T01:45:42.791 INFO:teuthology.orchestra.run.smithi083.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-11-16T01:45:42.816 INFO:teuthology.orchestra.run.smithi083.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-11-16T01:45:42.839 INFO:teuthology.orchestra.run.smithi083.stdout:Get:26 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 16.2.14-171-ged3c8e92-1focal [116 kB] 2023-11-16T01:45:42.842 INFO:teuthology.orchestra.run.smithi083.stdout:Get:27 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 16.2.14-171-ged3c8e92-1focal [448 kB] 2023-11-16T01:45:42.876 INFO:teuthology.orchestra.run.smithi083.stdout:Get:28 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 16.2.14-171-ged3c8e92-1focal [22.7 MB] 2023-11-16T01:45:42.933 INFO:teuthology.orchestra.run.smithi083.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-16T01:45:42.945 INFO:teuthology.orchestra.run.smithi083.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-11-16T01:45:42.949 INFO:teuthology.orchestra.run.smithi083.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-11-16T01:45:42.979 INFO:teuthology.orchestra.run.smithi136.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 16.2.14-171-ged3c8e92-1focal [116 kB] 2023-11-16T01:45:42.989 INFO:teuthology.orchestra.run.smithi136.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 16.2.14-171-ged3c8e92-1focal [448 kB] 2023-11-16T01:45:43.003 INFO:teuthology.orchestra.run.smithi136.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 16.2.14-171-ged3c8e92-1focal [22.7 MB] 2023-11-16T01:45:43.018 INFO:teuthology.orchestra.run.smithi083.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-11-16T01:45:43.033 INFO:teuthology.orchestra.run.smithi083.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-11-16T01:45:43.068 INFO:teuthology.orchestra.run.smithi083.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-11-16T01:45:43.094 INFO:teuthology.orchestra.run.smithi083.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-11-16T01:45:43.275 INFO:teuthology.orchestra.run.smithi083.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-11-16T01:45:43.806 INFO:teuthology.orchestra.run.smithi083.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-11-16T01:45:43.807 INFO:teuthology.orchestra.run.smithi083.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-11-16T01:45:43.810 INFO:teuthology.orchestra.run.smithi083.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-11-16T01:45:43.813 INFO:teuthology.orchestra.run.smithi083.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-11-16T01:45:43.825 INFO:teuthology.orchestra.run.smithi083.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-11-16T01:45:43.848 INFO:teuthology.orchestra.run.smithi083.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-11-16T01:45:43.861 INFO:teuthology.orchestra.run.smithi083.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-16T01:45:43.863 INFO:teuthology.orchestra.run.smithi083.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-11-16T01:45:43.899 INFO:teuthology.orchestra.run.smithi083.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-11-16T01:45:43.909 INFO:teuthology.orchestra.run.smithi083.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-11-16T01:45:44.084 INFO:teuthology.orchestra.run.smithi083.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-11-16T01:45:44.167 INFO:teuthology.orchestra.run.smithi083.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-11-16T01:45:44.302 INFO:teuthology.orchestra.run.smithi083.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-11-16T01:45:44.303 INFO:teuthology.orchestra.run.smithi083.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 16.2.14-171-ged3c8e92-1focal [5,919 kB] 2023-11-16T01:45:44.611 INFO:teuthology.orchestra.run.smithi083.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 16.2.14-171-ged3c8e92-1focal [204 kB] 2023-11-16T01:45:44.617 INFO:teuthology.orchestra.run.smithi083.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 16.2.14-171-ged3c8e92-1focal [126 kB] 2023-11-16T01:45:44.621 INFO:teuthology.orchestra.run.smithi083.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 16.2.14-171-ged3c8e92-1focal [1,550 kB] 2023-11-16T01:45:44.713 INFO:teuthology.orchestra.run.smithi136.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 16.2.14-171-ged3c8e92-1focal [5,919 kB] 2023-11-16T01:45:44.729 INFO:teuthology.orchestra.run.smithi083.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 16.2.14-171-ged3c8e92-1focal [6,996 kB] 2023-11-16T01:45:45.050 INFO:teuthology.orchestra.run.smithi136.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 16.2.14-171-ged3c8e92-1focal [204 kB] 2023-11-16T01:45:45.055 INFO:teuthology.orchestra.run.smithi136.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 16.2.14-171-ged3c8e92-1focal [126 kB] 2023-11-16T01:45:45.065 INFO:teuthology.orchestra.run.smithi083.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-11-16T01:45:45.086 INFO:teuthology.orchestra.run.smithi083.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-11-16T01:45:45.086 INFO:teuthology.orchestra.run.smithi136.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 16.2.14-171-ged3c8e92-1focal [1,550 kB] 2023-11-16T01:45:45.087 INFO:teuthology.orchestra.run.smithi083.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-11-16T01:45:45.144 INFO:teuthology.orchestra.run.smithi083.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 16.2.14-171-ged3c8e92-1focal [26.2 MB] 2023-11-16T01:45:45.163 INFO:teuthology.orchestra.run.smithi083.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-11-16T01:45:45.164 INFO:teuthology.orchestra.run.smithi083.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-11-16T01:45:45.164 INFO:teuthology.orchestra.run.smithi083.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-11-16T01:45:45.165 INFO:teuthology.orchestra.run.smithi083.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-11-16T01:45:45.165 INFO:teuthology.orchestra.run.smithi083.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-11-16T01:45:45.184 INFO:teuthology.orchestra.run.smithi136.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 16.2.14-171-ged3c8e92-1focal [6,996 kB] 2023-11-16T01:45:45.334 INFO:teuthology.orchestra.run.smithi083.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-11-16T01:45:45.418 INFO:teuthology.orchestra.run.smithi083.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-11-16T01:45:45.596 INFO:teuthology.orchestra.run.smithi083.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-11-16T01:45:45.611 INFO:teuthology.orchestra.run.smithi083.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-11-16T01:45:45.644 INFO:teuthology.orchestra.run.smithi083.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-11-16T01:45:45.680 INFO:teuthology.orchestra.run.smithi136.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 16.2.14-171-ged3c8e92-1focal [26.2 MB] 2023-11-16T01:45:45.773 INFO:teuthology.orchestra.run.smithi083.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-11-16T01:45:45.812 INFO:teuthology.orchestra.run.smithi083.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-11-16T01:45:45.833 INFO:teuthology.orchestra.run.smithi083.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-11-16T01:45:45.841 INFO:teuthology.orchestra.run.smithi083.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-11-16T01:45:45.883 INFO:teuthology.orchestra.run.smithi083.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-11-16T01:45:45.910 INFO:teuthology.orchestra.run.smithi083.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-11-16T01:45:46.097 INFO:teuthology.orchestra.run.smithi083.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-11-16T01:45:46.182 INFO:teuthology.orchestra.run.smithi083.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-11-16T01:45:46.283 INFO:teuthology.orchestra.run.smithi083.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-11-16T01:45:46.306 INFO:teuthology.orchestra.run.smithi083.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-11-16T01:45:46.420 INFO:teuthology.orchestra.run.smithi083.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-11-16T01:45:46.706 INFO:teuthology.orchestra.run.smithi083.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-11-16T01:45:46.707 INFO:teuthology.orchestra.run.smithi083.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-16T01:45:46.711 INFO:teuthology.orchestra.run.smithi083.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-11-16T01:45:46.713 INFO:teuthology.orchestra.run.smithi083.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-11-16T01:45:46.722 INFO:teuthology.orchestra.run.smithi083.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-11-16T01:45:46.726 INFO:teuthology.orchestra.run.smithi083.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-11-16T01:45:46.837 INFO:teuthology.orchestra.run.smithi083.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 16.2.14-171-ged3c8e92-1focal [4,060 B] 2023-11-16T01:45:46.837 INFO:teuthology.orchestra.run.smithi083.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 16.2.14-171-ged3c8e92-1focal [834 kB] 2023-11-16T01:45:46.846 INFO:teuthology.orchestra.run.smithi083.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 16.2.14-171-ged3c8e92-1focal [2,364 kB] 2023-11-16T01:45:46.892 INFO:teuthology.orchestra.run.smithi083.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-11-16T01:45:46.954 INFO:teuthology.orchestra.run.smithi083.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 16.2.14-171-ged3c8e92-1focal [77.4 kB] 2023-11-16T01:45:46.957 INFO:teuthology.orchestra.run.smithi083.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 16.2.14-171-ged3c8e92-1focal [100 kB] 2023-11-16T01:45:46.960 INFO:teuthology.orchestra.run.smithi083.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 16.2.14-171-ged3c8e92-1focal [2,824 kB] 2023-11-16T01:45:47.058 INFO:teuthology.orchestra.run.smithi083.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-11-16T01:45:47.115 INFO:teuthology.orchestra.run.smithi083.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 16.2.14-171-ged3c8e92-1focal [7,878 kB] 2023-11-16T01:45:47.311 INFO:teuthology.orchestra.run.smithi083.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-11-16T01:45:47.559 INFO:teuthology.orchestra.run.smithi083.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 16.2.14-171-ged3c8e92-1focal [4,176 B] 2023-11-16T01:45:47.559 INFO:teuthology.orchestra.run.smithi083.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 16.2.14-171-ged3c8e92-1focal [49.4 MB] 2023-11-16T01:45:47.903 INFO:teuthology.orchestra.run.smithi136.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 16.2.14-171-ged3c8e92-1focal [4,060 B] 2023-11-16T01:45:47.904 INFO:teuthology.orchestra.run.smithi136.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 16.2.14-171-ged3c8e92-1focal [834 kB] 2023-11-16T01:45:47.927 INFO:teuthology.orchestra.run.smithi136.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 16.2.14-171-ged3c8e92-1focal [2,364 kB] 2023-11-16T01:45:48.134 INFO:teuthology.orchestra.run.smithi136.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 16.2.14-171-ged3c8e92-1focal [77.4 kB] 2023-11-16T01:45:48.136 INFO:teuthology.orchestra.run.smithi136.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 16.2.14-171-ged3c8e92-1focal [100 kB] 2023-11-16T01:45:48.160 INFO:teuthology.orchestra.run.smithi136.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 16.2.14-171-ged3c8e92-1focal [2,824 kB] 2023-11-16T01:45:48.391 INFO:teuthology.orchestra.run.smithi136.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 16.2.14-171-ged3c8e92-1focal [7,878 kB] 2023-11-16T01:45:49.139 INFO:teuthology.orchestra.run.smithi136.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 16.2.14-171-ged3c8e92-1focal [4,176 B] 2023-11-16T01:45:49.139 INFO:teuthology.orchestra.run.smithi136.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 16.2.14-171-ged3c8e92-1focal [49.4 MB] 2023-11-16T01:45:50.788 INFO:teuthology.orchestra.run.smithi083.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 16.2.14-171-ged3c8e92-1focal [21.5 kB] 2023-11-16T01:45:50.788 INFO:teuthology.orchestra.run.smithi083.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 16.2.14-171-ged3c8e92-1focal [11.3 MB] 2023-11-16T01:45:51.317 INFO:teuthology.orchestra.run.smithi083.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 16.2.14-171-ged3c8e92-1focal [78.2 kB] 2023-11-16T01:45:51.936 INFO:teuthology.orchestra.run.smithi083.stdout:Fetched 174 MB in 10s (17.8 MB/s) 2023-11-16T01:45:52.022 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libjaeger. 2023-11-16T01:45:52.079 INFO:teuthology.orchestra.run.smithi083.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141728 files and directories currently installed.) 2023-11-16T01:45:52.083 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../00-libjaeger_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:52.128 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libjaeger (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:52.448 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-11-16T01:45:52.465 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../01-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-11-16T01:45:52.548 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-11-16T01:45:52.885 INFO:teuthology.orchestra.run.smithi136.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 16.2.14-171-ged3c8e92-1focal [21.5 kB] 2023-11-16T01:45:52.885 INFO:teuthology.orchestra.run.smithi136.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 16.2.14-171-ged3c8e92-1focal [11.3 MB] 2023-11-16T01:45:52.935 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../02-librbd1_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:53.060 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking librbd1 (16.2.14-171-ged3c8e92-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-11-16T01:45:53.237 INFO:teuthology.orchestra.run.smithi136.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-14-1227-pacific/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 16.2.14-171-ged3c8e92-1focal [78.2 kB] 2023-11-16T01:45:53.760 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../03-librados2_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:53.862 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 174 MB in 12s (15.0 MB/s) 2023-11-16T01:45:53.882 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking librados2 (16.2.14-171-ged3c8e92-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-11-16T01:45:53.947 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libjaeger. 2023-11-16T01:45:54.003 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141728 files and directories currently installed.) 2023-11-16T01:45:54.007 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../00-libjaeger_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:54.045 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libjaeger (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:54.306 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-11-16T01:45:54.324 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../01-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-11-16T01:45:54.407 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-11-16T01:45:54.561 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libcephfs2. 2023-11-16T01:45:54.580 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../04-libcephfs2_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:54.619 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libcephfs2 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:54.811 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../02-librbd1_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:54.905 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-rados. 2023-11-16T01:45:54.920 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../05-python3-rados_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:54.930 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librbd1 (16.2.14-171-ged3c8e92-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-11-16T01:45:54.955 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-rados (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:55.315 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-16T01:45:55.331 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../06-python3-ceph-argparse_16.2.14-171-ged3c8e92-1focal_all.deb ... 2023-11-16T01:45:55.374 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-ceph-argparse (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:55.635 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../03-librados2_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:55.702 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-cephfs. 2023-11-16T01:45:55.718 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../07-python3-cephfs_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:55.758 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librados2 (16.2.14-171-ged3c8e92-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-11-16T01:45:55.760 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-cephfs (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:56.071 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-16T01:45:56.089 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../08-python3-ceph-common_16.2.14-171-ged3c8e92-1focal_all.deb ... 2023-11-16T01:45:56.129 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-ceph-common (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:56.440 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-prettytable. 2023-11-16T01:45:56.460 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../09-python3-prettytable_0.7.2-5_all.deb ... 2023-11-16T01:45:56.462 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libcephfs2. 2023-11-16T01:45:56.478 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../04-libcephfs2_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:56.506 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-11-16T01:45:56.520 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libcephfs2 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:56.736 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-rbd. 2023-11-16T01:45:56.755 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../10-python3-rbd_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:56.794 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-rbd (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:56.848 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rados. 2023-11-16T01:45:56.865 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../05-python3-rados_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:56.906 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rados (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:57.147 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-11-16T01:45:57.165 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../11-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-11-16T01:45:57.230 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-11-16T01:45:57.251 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-16T01:45:57.269 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../06-python3-ceph-argparse_16.2.14-171-ged3c8e92-1focal_all.deb ... 2023-11-16T01:45:57.309 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-ceph-argparse (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:57.533 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-11-16T01:45:57.549 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../12-librabbitmq4_0.10.0-1_amd64.deb ... 2023-11-16T01:45:57.579 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cephfs. 2023-11-16T01:45:57.583 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-11-16T01:45:57.595 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../07-python3-cephfs_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:57.637 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cephfs (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:57.921 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-16T01:45:57.938 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../13-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-11-16T01:45:57.956 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-16T01:45:57.976 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../08-python3-ceph-common_16.2.14-171-ged3c8e92-1focal_all.deb ... 2023-11-16T01:45:57.979 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-11-16T01:45:58.014 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-ceph-common (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:58.343 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-prettytable. 2023-11-16T01:45:58.349 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package librgw2. 2023-11-16T01:45:58.359 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../09-python3-prettytable_0.7.2-5_all.deb ... 2023-11-16T01:45:58.368 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../14-librgw2_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:58.401 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-11-16T01:45:58.407 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking librgw2 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:58.630 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rbd. 2023-11-16T01:45:58.649 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../10-python3-rbd_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:58.680 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rbd (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:59.035 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-rgw. 2023-11-16T01:45:59.053 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../15-python3-rgw_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:45:59.093 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-rgw (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:45:59.140 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-11-16T01:45:59.160 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../11-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-11-16T01:45:59.274 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-11-16T01:45:59.413 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libleveldb1d:amd64. 2023-11-16T01:45:59.432 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../16-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2023-11-16T01:45:59.529 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2023-11-16T01:45:59.644 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-11-16T01:45:59.660 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../12-librabbitmq4_0.10.0-1_amd64.deb ... 2023-11-16T01:45:59.703 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-11-16T01:46:00.015 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-16T01:46:00.031 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../13-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-11-16T01:46:00.049 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package liboath0. 2023-11-16T01:46:00.062 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../17-liboath0_2.6.1-1.3_amd64.deb ... 2023-11-16T01:46:00.064 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-11-16T01:46:00.099 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-11-16T01:46:00.392 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librgw2. 2023-11-16T01:46:00.402 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libradosstriper1. 2023-11-16T01:46:00.408 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../14-librgw2_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:00.418 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../18-libradosstriper1_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:00.450 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librgw2 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:00.460 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libradosstriper1 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:00.947 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-common. 2023-11-16T01:46:00.967 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../19-ceph-common_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:01.005 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-common (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:01.053 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rgw. 2023-11-16T01:46:01.070 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../15-python3-rgw_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:01.111 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rgw (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:01.430 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libleveldb1d:amd64. 2023-11-16T01:46:01.446 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../16-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2023-11-16T01:46:01.489 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2023-11-16T01:46:01.833 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liboath0. 2023-11-16T01:46:01.851 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../17-liboath0_2.6.1-1.3_amd64.deb ... 2023-11-16T01:46:01.883 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-11-16T01:46:02.170 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libradosstriper1. 2023-11-16T01:46:02.186 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../18-libradosstriper1_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:02.228 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libradosstriper1 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:02.715 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-common. 2023-11-16T01:46:02.731 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../19-ceph-common_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:02.773 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-common (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:03.161 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-base. 2023-11-16T01:46:03.178 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../20-ceph-base_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:03.221 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-base (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:04.174 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-cryptography. 2023-11-16T01:46:04.190 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../21-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-11-16T01:46:04.232 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-11-16T01:46:04.627 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-openssl. 2023-11-16T01:46:04.647 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../22-python3-openssl_19.0.0-1build1_all.deb ... 2023-11-16T01:46:04.685 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-11-16T01:46:04.938 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-16T01:46:04.956 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../23-ceph-mgr-modules-core_16.2.14-171-ged3c8e92-1focal_all.deb ... 2023-11-16T01:46:04.996 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr-modules-core (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:05.054 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-base. 2023-11-16T01:46:05.071 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../20-ceph-base_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:05.123 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-base (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:05.393 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-16T01:46:05.410 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../24-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-11-16T01:46:05.451 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-11-16T01:46:05.754 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-16T01:46:05.770 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../25-python3-cherrypy3_8.9.1-6_all.deb ... 2023-11-16T01:46:05.812 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-11-16T01:46:06.077 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cryptography. 2023-11-16T01:46:06.091 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../21-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-11-16T01:46:06.133 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-11-16T01:46:06.157 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-jwt. 2023-11-16T01:46:06.174 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../26-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-11-16T01:46:06.224 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-11-16T01:46:06.512 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-openssl. 2023-11-16T01:46:06.527 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-logutils. 2023-11-16T01:46:06.528 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../22-python3-openssl_19.0.0-1build1_all.deb ... 2023-11-16T01:46:06.543 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../27-python3-logutils_0.3.3-7_all.deb ... 2023-11-16T01:46:06.562 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-11-16T01:46:06.585 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-11-16T01:46:06.865 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-16T01:46:06.881 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../23-ceph-mgr-modules-core_16.2.14-171-ged3c8e92-1focal_all.deb ... 2023-11-16T01:46:06.888 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-16T01:46:06.905 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../28-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-11-16T01:46:06.923 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-modules-core (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:06.947 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-11-16T01:46:07.274 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-mako. 2023-11-16T01:46:07.293 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../29-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-11-16T01:46:07.304 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-16T01:46:07.323 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../24-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-11-16T01:46:07.333 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-11-16T01:46:07.361 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-11-16T01:46:07.639 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-16T01:46:07.644 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-16T01:46:07.659 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../25-python3-cherrypy3_8.9.1-6_all.deb ... 2023-11-16T01:46:07.663 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../30-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-16T01:46:07.689 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-11-16T01:46:07.703 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-16T01:46:08.039 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-16T01:46:08.043 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jwt. 2023-11-16T01:46:08.057 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../31-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-11-16T01:46:08.059 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../26-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-11-16T01:46:08.097 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-11-16T01:46:08.101 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-11-16T01:46:08.396 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-logutils. 2023-11-16T01:46:08.400 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-webob. 2023-11-16T01:46:08.415 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../27-python3-logutils_0.3.3-7_all.deb ... 2023-11-16T01:46:08.417 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../32-python3-webob_1%3a1.8.5-2_all.deb ... 2023-11-16T01:46:08.458 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-11-16T01:46:08.488 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-11-16T01:46:08.786 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-soupsieve. 2023-11-16T01:46:08.800 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-16T01:46:08.802 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../33-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-11-16T01:46:08.815 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../28-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-11-16T01:46:08.844 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-11-16T01:46:08.857 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-11-16T01:46:09.160 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-mako. 2023-11-16T01:46:09.178 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../29-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-11-16T01:46:09.189 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-bs4. 2023-11-16T01:46:09.209 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../34-python3-bs4_4.8.2-1_all.deb ... 2023-11-16T01:46:09.219 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-11-16T01:46:09.247 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-11-16T01:46:09.589 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-16T01:46:09.600 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-waitress. 2023-11-16T01:46:09.606 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../30-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-16T01:46:09.617 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../35-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-11-16T01:46:09.655 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-16T01:46:09.661 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-11-16T01:46:09.950 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-16T01:46:09.966 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../31-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-11-16T01:46:10.008 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-11-16T01:46:10.036 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-tempita. 2023-11-16T01:46:10.052 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../36-python3-tempita_0.5.2-6_all.deb ... 2023-11-16T01:46:10.103 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-11-16T01:46:10.336 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-webob. 2023-11-16T01:46:10.352 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../32-python3-webob_1%3a1.8.5-2_all.deb ... 2023-11-16T01:46:10.395 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-11-16T01:46:10.406 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-paste. 2023-11-16T01:46:10.422 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../37-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-11-16T01:46:10.464 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-11-16T01:46:10.723 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-soupsieve. 2023-11-16T01:46:10.738 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../33-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-11-16T01:46:10.781 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-11-16T01:46:10.792 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-16T01:46:10.811 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../38-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-11-16T01:46:10.850 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-11-16T01:46:11.109 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-bs4. 2023-11-16T01:46:11.128 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../34-python3-bs4_4.8.2-1_all.deb ... 2023-11-16T01:46:11.153 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-16T01:46:11.167 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-11-16T01:46:11.170 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../39-python3-pastedeploy_2.1.0-1_all.deb ... 2023-11-16T01:46:11.220 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-11-16T01:46:11.520 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-waitress. 2023-11-16T01:46:11.523 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-webtest. 2023-11-16T01:46:11.538 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../35-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-11-16T01:46:11.541 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../40-python3-webtest_2.0.32-2_all.deb ... 2023-11-16T01:46:11.581 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-11-16T01:46:11.582 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-11-16T01:46:11.865 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-tempita. 2023-11-16T01:46:11.882 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../36-python3-tempita_0.5.2-6_all.deb ... 2023-11-16T01:46:11.893 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pecan. 2023-11-16T01:46:11.912 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../41-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-11-16T01:46:11.923 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-11-16T01:46:11.950 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-11-16T01:46:12.260 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-paste. 2023-11-16T01:46:12.278 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../37-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-11-16T01:46:12.306 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-16T01:46:12.318 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-11-16T01:46:12.323 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../42-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-11-16T01:46:12.364 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-11-16T01:46:12.613 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-16T01:46:12.632 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../38-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-11-16T01:46:12.671 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-11-16T01:46:12.734 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-16T01:46:12.752 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../43-libsqlite3-mod-ceph_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:12.792 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libsqlite3-mod-ceph (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:13.016 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-16T01:46:13.033 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../39-python3-pastedeploy_2.1.0-1_all.deb ... 2023-11-16T01:46:13.074 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-11-16T01:46:13.137 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr. 2023-11-16T01:46:13.153 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../44-ceph-mgr_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:13.195 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:13.361 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-webtest. 2023-11-16T01:46:13.380 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../40-python3-webtest_2.0.32-2_all.deb ... 2023-11-16T01:46:13.419 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-11-16T01:46:13.715 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mon. 2023-11-16T01:46:13.731 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../45-ceph-mon_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:13.747 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pecan. 2023-11-16T01:46:13.765 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../41-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-11-16T01:46:13.773 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mon (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:13.805 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-11-16T01:46:14.146 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-16T01:46:14.159 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../42-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-11-16T01:46:14.202 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-11-16T01:46:14.547 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-16T01:46:14.563 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../43-libsqlite3-mod-ceph_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:14.877 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-osd. 2023-11-16T01:46:14.882 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libsqlite3-mod-ceph (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:14.893 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../46-ceph-osd_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:14.935 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-osd (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:15.709 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr. 2023-11-16T01:46:15.726 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../44-ceph-mgr_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:15.758 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:16.170 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mon. 2023-11-16T01:46:16.188 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../45-ceph-mon_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:16.228 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mon (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:17.307 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-osd. 2023-11-16T01:46:17.325 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../46-ceph-osd_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:17.367 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-osd (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:17.966 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph. 2023-11-16T01:46:17.983 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../47-ceph_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:18.024 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:18.360 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-fuse. 2023-11-16T01:46:18.376 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../48-ceph-fuse_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:18.418 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-fuse (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:18.821 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mds. 2023-11-16T01:46:18.841 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../49-ceph-mds_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:18.880 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mds (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:19.449 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package cephadm. 2023-11-16T01:46:19.466 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../50-cephadm_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:19.508 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking cephadm (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:19.827 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-jinja2. 2023-11-16T01:46:19.843 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../51-python3-jinja2_2.10.1-2_all.deb ... 2023-11-16T01:46:19.885 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-11-16T01:46:20.054 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph. 2023-11-16T01:46:20.073 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../47-ceph_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:20.113 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:20.230 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-16T01:46:20.247 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../52-ceph-mgr-cephadm_16.2.14-171-ged3c8e92-1focal_all.deb ... 2023-11-16T01:46:20.288 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr-cephadm (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:20.432 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-fuse. 2023-11-16T01:46:20.448 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../48-ceph-fuse_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:20.490 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-fuse (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:20.641 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-16T01:46:20.658 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../53-python3-repoze.lru_0.7-2_all.deb ... 2023-11-16T01:46:20.700 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-16T01:46:20.944 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mds. 2023-11-16T01:46:20.960 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../49-ceph-mds_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:20.977 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libjs-underscore. 2023-11-16T01:46:20.994 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../54-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-11-16T01:46:21.002 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mds (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:21.035 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-11-16T01:46:21.321 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-11-16T01:46:21.338 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../55-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-11-16T01:46:21.379 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-11-16T01:46:21.522 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package cephadm. 2023-11-16T01:46:21.539 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../50-cephadm_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:21.598 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking cephadm (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:21.699 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-routes. 2023-11-16T01:46:21.715 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../56-python3-routes_2.4.1-2_all.deb ... 2023-11-16T01:46:21.757 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-11-16T01:46:21.925 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jinja2. 2023-11-16T01:46:21.942 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../51-python3-jinja2_2.10.1-2_all.deb ... 2023-11-16T01:46:21.983 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-11-16T01:46:22.127 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-16T01:46:22.143 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../57-ceph-mgr-dashboard_16.2.14-171-ged3c8e92-1focal_all.deb ... 2023-11-16T01:46:22.185 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr-dashboard (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:22.311 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-16T01:46:22.330 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../52-ceph-mgr-cephadm_16.2.14-171-ged3c8e92-1focal_all.deb ... 2023-11-16T01:46:22.369 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-cephadm (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:22.723 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-16T01:46:22.739 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../53-python3-repoze.lru_0.7-2_all.deb ... 2023-11-16T01:46:22.781 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-16T01:46:23.033 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libjs-underscore. 2023-11-16T01:46:23.052 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../54-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-11-16T01:46:23.091 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-11-16T01:46:23.168 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-decorator. 2023-11-16T01:46:23.188 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../58-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-11-16T01:46:23.226 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-11-16T01:46:23.394 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-11-16T01:46:23.410 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../55-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-11-16T01:46:23.452 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-11-16T01:46:23.487 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-11-16T01:46:23.506 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../59-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-11-16T01:46:23.537 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-11-16T01:46:23.772 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-routes. 2023-11-16T01:46:23.788 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../56-python3-routes_2.4.1-2_all.deb ... 2023-11-16T01:46:23.830 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-11-16T01:46:23.882 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-scipy. 2023-11-16T01:46:23.898 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../60-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-11-16T01:46:23.940 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-11-16T01:46:24.166 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-16T01:46:24.183 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../57-ceph-mgr-dashboard_16.2.14-171-ged3c8e92-1focal_all.deb ... 2023-11-16T01:46:24.233 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-dashboard (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:25.192 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-decorator. 2023-11-16T01:46:25.212 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../58-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-11-16T01:46:25.250 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-11-16T01:46:25.299 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-11-16T01:46:25.318 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../61-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-11-16T01:46:25.357 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-11-16T01:46:25.555 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-11-16T01:46:25.571 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../59-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-11-16T01:46:25.652 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-11-16T01:46:25.919 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-joblib. 2023-11-16T01:46:25.937 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../62-python3-joblib_0.14.0-3_all.deb ... 2023-11-16T01:46:25.974 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-scipy. 2023-11-16T01:46:25.977 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-11-16T01:46:25.991 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../60-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-11-16T01:46:26.024 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-11-16T01:46:26.431 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-sklearn. 2023-11-16T01:46:26.449 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../63-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-11-16T01:46:26.490 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-11-16T01:46:27.197 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-16T01:46:27.214 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../64-ceph-mgr-diskprediction-local_16.2.14-171-ged3c8e92-1focal_all.deb ... 2023-11-16T01:46:27.272 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr-diskprediction-local (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:27.292 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-11-16T01:46:27.310 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../61-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-11-16T01:46:27.350 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-11-16T01:46:27.913 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-joblib. 2023-11-16T01:46:37.606 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-cachetools. 2023-11-16T01:46:37.643 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../65-python3-cachetools_4.0.0-1_all.deb ... 2023-11-16T01:46:37.643 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-11-16T01:46:37.643 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-16T01:46:37.643 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../66-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-11-16T01:46:37.643 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-11-16T01:46:37.643 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-16T01:46:37.643 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../67-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-11-16T01:46:37.643 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-11-16T01:46:37.643 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-rsa. 2023-11-16T01:46:37.643 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../68-python3-rsa_4.0-3_all.deb ... 2023-11-16T01:46:37.643 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-rsa (4.0-3) ... 2023-11-16T01:46:37.644 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-google-auth. 2023-11-16T01:46:37.644 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../69-python3-google-auth_1.5.1-2_all.deb ... 2023-11-16T01:46:37.644 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-11-16T01:46:37.644 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-blinker. 2023-11-16T01:46:37.644 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../70-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-11-16T01:46:37.644 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-11-16T01:46:37.644 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-oauthlib. 2023-11-16T01:46:37.644 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../71-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-11-16T01:46:37.644 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-11-16T01:46:37.644 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-16T01:46:37.644 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../72-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-11-16T01:46:37.644 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-11-16T01:46:37.644 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-websocket. 2023-11-16T01:46:37.645 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../73-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-11-16T01:46:37.645 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-11-16T01:46:37.645 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-16T01:46:37.645 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../74-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-11-16T01:46:37.645 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-11-16T01:46:37.645 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-16T01:46:37.645 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../75-ceph-mgr-k8sevents_16.2.14-171-ged3c8e92-1focal_all.deb ... 2023-11-16T01:46:37.645 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr-k8sevents (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:37.645 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-16T01:46:37.645 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../76-libonig5_6.9.4-1_amd64.deb ... 2023-11-16T01:46:37.645 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-11-16T01:46:37.645 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-16T01:46:37.645 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../77-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-11-16T01:46:37.646 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-11-16T01:46:37.646 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package jq. 2023-11-16T01:46:37.646 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../78-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-11-16T01:46:37.646 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-11-16T01:46:37.646 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package socat. 2023-11-16T01:46:37.646 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../79-socat_1.7.3.3-2_amd64.deb ... 2023-11-16T01:46:37.646 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking socat (1.7.3.3-2) ... 2023-11-16T01:46:37.646 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package xmlstarlet. 2023-11-16T01:46:37.646 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../80-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-11-16T01:46:37.646 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-11-16T01:46:37.646 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-test. 2023-11-16T01:46:37.646 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../81-ceph-test_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:37.647 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-test (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:37.648 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../62-python3-joblib_0.14.0-3_all.deb ... 2023-11-16T01:46:37.648 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-11-16T01:46:37.648 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-sklearn. 2023-11-16T01:46:37.648 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../63-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-11-16T01:46:37.648 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-11-16T01:46:37.648 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-16T01:46:37.648 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../64-ceph-mgr-diskprediction-local_16.2.14-171-ged3c8e92-1focal_all.deb ... 2023-11-16T01:46:37.648 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-diskprediction-local (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:37.648 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cachetools. 2023-11-16T01:46:37.648 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../65-python3-cachetools_4.0.0-1_all.deb ... 2023-11-16T01:46:37.648 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-11-16T01:46:37.648 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-16T01:46:37.648 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../66-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-11-16T01:46:37.649 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-11-16T01:46:37.649 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-16T01:46:37.649 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../67-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-11-16T01:46:37.649 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-11-16T01:46:37.649 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rsa. 2023-11-16T01:46:37.649 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../68-python3-rsa_4.0-3_all.deb ... 2023-11-16T01:46:37.650 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rsa (4.0-3) ... 2023-11-16T01:46:37.650 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-google-auth. 2023-11-16T01:46:37.650 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../69-python3-google-auth_1.5.1-2_all.deb ... 2023-11-16T01:46:37.650 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-11-16T01:46:37.650 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-blinker. 2023-11-16T01:46:37.650 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../70-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-11-16T01:46:37.650 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-11-16T01:46:37.651 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-oauthlib. 2023-11-16T01:46:37.651 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../71-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-11-16T01:46:37.651 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-11-16T01:46:37.651 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-16T01:46:37.651 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../72-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-11-16T01:46:37.651 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-11-16T01:46:37.651 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-websocket. 2023-11-16T01:46:37.651 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../73-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-11-16T01:46:37.651 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-11-16T01:46:37.688 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-16T01:46:37.688 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../74-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-11-16T01:46:37.688 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-11-16T01:46:37.688 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-16T01:46:37.688 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../75-ceph-mgr-k8sevents_16.2.14-171-ged3c8e92-1focal_all.deb ... 2023-11-16T01:46:37.688 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-k8sevents (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:37.688 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-16T01:46:37.688 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../76-libonig5_6.9.4-1_amd64.deb ... 2023-11-16T01:46:37.689 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-11-16T01:46:37.689 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-16T01:46:37.689 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../77-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-11-16T01:46:37.689 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-11-16T01:46:37.689 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package jq. 2023-11-16T01:46:37.689 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../78-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-11-16T01:46:37.689 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-11-16T01:46:37.689 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package socat. 2023-11-16T01:46:37.689 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../79-socat_1.7.3.3-2_amd64.deb ... 2023-11-16T01:46:37.689 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking socat (1.7.3.3-2) ... 2023-11-16T01:46:37.689 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package xmlstarlet. 2023-11-16T01:46:37.689 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../80-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-11-16T01:46:37.689 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-11-16T01:46:37.689 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-test. 2023-11-16T01:46:37.690 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../81-ceph-test_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:37.690 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-test (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:39.442 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-16T01:46:39.458 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../82-libcephfs-dev_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:39.500 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libcephfs-dev (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:39.928 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package nvme-cli. 2023-11-16T01:46:39.946 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../83-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-11-16T01:46:40.005 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-11-16T01:46:40.352 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-atomicwrites. 2023-11-16T01:46:40.369 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../84-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-11-16T01:46:40.434 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-11-16T01:46:40.671 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-attr. 2023-11-16T01:46:40.692 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../85-python3-attr_19.3.0-2_all.deb ... 2023-11-16T01:46:40.729 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-11-16T01:46:41.073 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-webencodings. 2023-11-16T01:46:41.091 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../86-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-11-16T01:46:41.132 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-11-16T01:46:41.405 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-16T01:46:41.421 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../82-libcephfs-dev_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:41.434 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-html5lib. 2023-11-16T01:46:41.451 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../87-python3-html5lib_1.0.1-2_all.deb ... 2023-11-16T01:46:41.462 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libcephfs-dev (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:41.492 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-11-16T01:46:41.871 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-11-16T01:46:41.883 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package nvme-cli. 2023-11-16T01:46:41.889 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../88-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-11-16T01:46:41.899 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../83-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-11-16T01:46:41.929 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-11-16T01:46:41.959 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-11-16T01:46:42.298 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-atomicwrites. 2023-11-16T01:46:42.315 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../84-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-11-16T01:46:42.365 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-11-16T01:46:42.391 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-packaging. 2023-11-16T01:46:42.409 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../89-python3-packaging_20.3-1_all.deb ... 2023-11-16T01:46:42.448 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-packaging (20.3-1) ... 2023-11-16T01:46:42.684 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-attr. 2023-11-16T01:46:42.704 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../85-python3-attr_19.3.0-2_all.deb ... 2023-11-16T01:46:42.742 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-11-16T01:46:42.785 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pastescript. 2023-11-16T01:46:42.802 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../90-python3-pastescript_2.0.2-4_all.deb ... 2023-11-16T01:46:42.843 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-16T01:46:43.070 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-webencodings. 2023-11-16T01:46:43.087 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../86-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-11-16T01:46:43.128 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-11-16T01:46:43.163 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pluggy. 2023-11-16T01:46:43.180 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../91-python3-pluggy_0.13.0-2_all.deb ... 2023-11-16T01:46:43.221 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-11-16T01:46:43.456 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-html5lib. 2023-11-16T01:46:43.477 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../87-python3-html5lib_1.0.1-2_all.deb ... 2023-11-16T01:46:43.514 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-11-16T01:46:43.516 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-psutil. 2023-11-16T01:46:43.536 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../92-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-11-16T01:46:43.600 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-11-16T01:46:43.851 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-11-16T01:46:43.869 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../88-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-11-16T01:46:43.900 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-11-16T01:46:43.912 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-py. 2023-11-16T01:46:43.932 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../93-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-11-16T01:46:43.970 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-11-16T01:46:44.290 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-16T01:46:44.310 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../94-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-11-16T01:46:44.348 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-11-16T01:46:44.387 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-packaging. 2023-11-16T01:46:44.408 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../89-python3-packaging_20.3-1_all.deb ... 2023-11-16T01:46:44.445 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-packaging (20.3-1) ... 2023-11-16T01:46:44.693 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-16T01:46:44.710 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../95-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-11-16T01:46:44.751 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-11-16T01:46:44.758 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pastescript. 2023-11-16T01:46:44.774 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../90-python3-pastescript_2.0.2-4_all.deb ... 2023-11-16T01:46:44.815 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-16T01:46:45.053 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pytest. 2023-11-16T01:46:45.070 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../96-python3-pytest_4.6.9-1_all.deb ... 2023-11-16T01:46:45.120 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-11-16T01:46:45.169 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pluggy. 2023-11-16T01:46:45.186 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../91-python3-pluggy_0.13.0-2_all.deb ... 2023-11-16T01:46:45.226 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-11-16T01:46:45.507 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-simplejson. 2023-11-16T01:46:45.525 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../97-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-11-16T01:46:45.555 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-psutil. 2023-11-16T01:46:45.565 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-11-16T01:46:45.573 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../92-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-11-16T01:46:45.613 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-11-16T01:46:45.993 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-py. 2023-11-16T01:46:46.002 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package radosgw. 2023-11-16T01:46:46.010 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../93-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-11-16T01:46:46.018 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../98-radosgw_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:46.051 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-11-16T01:46:46.060 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking radosgw (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:46.379 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-16T01:46:46.396 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../94-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-11-16T01:46:46.437 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-11-16T01:46:46.749 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-16T01:46:46.766 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../95-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-11-16T01:46:46.832 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-11-16T01:46:47.135 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pytest. 2023-11-16T01:46:47.151 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../96-python3-pytest_4.6.9-1_all.deb ... 2023-11-16T01:46:47.193 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-11-16T01:46:47.280 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package rbd-fuse. 2023-11-16T01:46:47.298 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../99-rbd-fuse_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:47.338 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking rbd-fuse (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:47.547 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-simplejson. 2023-11-16T01:46:47.565 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../97-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-11-16T01:46:47.605 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-11-16T01:46:47.634 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-attr (19.3.0-2) ... 2023-11-16T01:46:47.926 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2023-11-16T01:46:48.052 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-11-16T01:46:48.058 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package radosgw. 2023-11-16T01:46:48.075 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../98-radosgw_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:48.116 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking radosgw (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:48.195 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-11-16T01:46:48.388 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up cephadm (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:48.959 INFO:teuthology.orchestra.run.smithi083.stdout:Adding system user cephadm....done 2023-11-16T01:46:49.136 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-11-16T01:46:49.312 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package rbd-fuse. 2023-11-16T01:46:49.331 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../99-rbd-fuse_16.2.14-171-ged3c8e92-1focal_amd64.deb ... 2023-11-16T01:46:49.370 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking rbd-fuse (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:49.491 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-16T01:46:49.692 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-attr (19.3.0-2) ... 2023-11-16T01:46:49.806 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-11-16T01:46:49.957 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-11-16T01:46:49.991 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2023-11-16T01:46:50.117 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-11-16T01:46:50.243 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-11-16T01:46:50.258 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-11-16T01:46:50.453 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up cephadm (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:50.654 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-11-16T01:46:50.938 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-11-16T01:46:50.948 INFO:teuthology.orchestra.run.smithi136.stdout:Adding system user cephadm....done 2023-11-16T01:46:51.101 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-11-16T01:46:51.222 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-11-16T01:46:51.486 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-16T01:46:51.506 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-11-16T01:46:51.774 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-ceph-argparse (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:51.779 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-11-16T01:46:51.897 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-11-16T01:46:52.034 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-11-16T01:46:52.160 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-11-16T01:46:52.207 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-11-16T01:46:52.445 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-11-16T01:46:52.618 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-11-16T01:46:52.729 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-11-16T01:46:52.911 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-11-16T01:46:53.106 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-11-16T01:46:53.204 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-11-16T01:46:53.382 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-11-16T01:46:53.489 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-11-16T01:46:53.709 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-11-16T01:46:53.757 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-ceph-argparse (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:53.985 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libjaeger (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:54.041 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-11-16T01:46:54.110 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-11-16T01:46:54.167 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-11-16T01:46:54.445 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-11-16T01:46:54.452 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-11-16T01:46:54.721 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-packaging (20.3-1) ... 2023-11-16T01:46:54.736 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-11-16T01:46:55.008 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-11-16T01:46:55.130 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-11-16T01:46:55.135 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-11-16T01:46:55.390 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-11-16T01:46:55.428 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-11-16T01:46:55.749 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-11-16T01:46:55.813 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-11-16T01:46:56.042 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libjaeger (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:56.165 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-11-16T01:46:56.170 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-11-16T01:46:56.511 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-11-16T01:46:56.542 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up socat (1.7.3.3-2) ... 2023-11-16T01:46:56.651 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-ceph-common (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:56.837 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-packaging (20.3-1) ... 2023-11-16T01:46:56.954 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-11-16T01:46:57.130 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-11-16T01:46:57.268 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-11-16T01:46:57.279 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-11-16T01:46:57.405 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-11-16T01:46:57.531 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-11-16T01:46:57.562 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-11-16T01:46:57.808 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-11-16T01:46:57.963 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-11-16T01:46:58.101 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-11-16T01:46:58.227 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-11-16T01:46:58.340 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-11-16T01:46:58.546 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-11-16T01:46:58.700 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up socat (1.7.3.3-2) ... 2023-11-16T01:46:58.826 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-ceph-common (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:46:58.834 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-11-16T01:46:59.110 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-11-16T01:46:59.177 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-11-16T01:46:59.239 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-16T01:46:59.521 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-11-16T01:46:59.521 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-11-16T01:46:59.639 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-11-16T01:46:59.740 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-11-16T01:46:59.815 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-11-16T01:47:00.042 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-11-16T01:47:00.124 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-11-16T01:47:00.360 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-11-16T01:47:00.465 INFO:teuthology.orchestra.run.smithi083.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-11-16T01:47:00.486 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-11-16T01:47:00.718 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-11-16T01:47:00.789 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-11-16T01:47:00.863 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-11-16T01:47:01.057 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-11-16T01:47:01.181 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-11-16T01:47:01.361 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-11-16T01:47:01.510 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-16T01:47:01.532 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-11-16T01:47:01.683 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-11-16T01:47:01.809 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-11-16T01:47:02.136 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-11-16T01:47:02.171 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-11-16T01:47:02.457 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-11-16T01:47:02.703 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up librados2 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:02.749 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-11-16T01:47:02.829 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-11-16T01:47:03.134 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-11-16T01:47:03.384 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-11-16T01:47:03.510 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-11-16T01:47:03.839 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-11-16T01:47:04.190 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-11-16T01:47:04.285 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up librgw2 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:04.308 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-11-16T01:47:04.411 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libsqlite3-mod-ceph (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:04.434 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-11-16T01:47:04.570 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-11-16T01:47:04.713 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-11-16T01:47:04.776 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-11-16T01:47:05.066 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-11-16T01:47:05.078 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librados2 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:05.204 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-11-16T01:47:05.365 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-11-16T01:47:05.683 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-11-16T01:47:06.026 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libcephfs2 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:06.144 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-11-16T01:47:06.462 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libradosstriper1 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:06.589 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-11-16T01:47:06.659 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librgw2 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:06.801 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libsqlite3-mod-ceph (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:06.923 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-11-16T01:47:06.927 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-11-16T01:47:07.053 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-11-16T01:47:07.447 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-11-16T01:47:07.739 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-11-16T01:47:08.023 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up librbd1 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:08.024 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-11-16T01:47:08.166 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-11-16T01:47:08.292 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-11-16T01:47:08.350 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libcephfs2 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:08.418 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-rsa (4.0-3) ... 2023-11-16T01:47:08.501 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-11-16T01:47:08.694 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-11-16T01:47:08.836 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libradosstriper1 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:08.962 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-11-16T01:47:08.998 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr-modules-core (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:09.141 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-fuse (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:09.305 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-11-16T01:47:09.368 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-16T01:47:09.368 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-16T01:47:09.801 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libcephfs-dev (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:09.930 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-11-16T01:47:10.237 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-16T01:47:10.347 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librbd1 (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:10.456 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-11-16T01:47:10.557 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-11-16T01:47:10.571 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-11-16T01:47:10.674 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rsa (4.0-3) ... 2023-11-16T01:47:10.934 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-rados (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:10.986 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-11-16T01:47:11.068 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-rbd (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:11.194 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up rbd-fuse (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:11.304 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-modules-core (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:11.320 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-routes (2.4.1-2) ... 2023-11-16T01:47:11.439 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-fuse (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:11.621 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-11-16T01:47:11.640 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-16T01:47:11.640 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-16T01:47:12.008 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-rgw (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:12.098 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libcephfs-dev (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:12.134 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-cephfs (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:12.224 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-11-16T01:47:12.438 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-11-16T01:47:12.492 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-16T01:47:12.785 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-11-16T01:47:13.157 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rados (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:13.282 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rbd (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:13.348 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-common (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:13.400 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up rbd-fuse (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:13.500 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-routes (2.4.1-2) ... 2023-11-16T01:47:13.793 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-11-16T01:47:13.909 INFO:teuthology.orchestra.run.smithi083.stdout:Adding group ceph....done 2023-11-16T01:47:14.172 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rgw (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:14.269 INFO:teuthology.orchestra.run.smithi083.stdout:Adding system user ceph....done 2023-11-16T01:47:14.298 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cephfs (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:14.412 INFO:teuthology.orchestra.run.smithi083.stdout:Setting system user ceph properties....done 2023-11-16T01:47:14.439 INFO:teuthology.orchestra.run.smithi083.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-16T01:47:14.579 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-16T01:47:14.590 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-11-16T01:47:14.915 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-16T01:47:15.366 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-test (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:15.510 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-common (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:15.527 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up radosgw (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:15.805 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-16T01:47:15.806 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-16T01:47:16.047 INFO:teuthology.orchestra.run.smithi136.stdout:Adding group ceph....done 2023-11-16T01:47:16.465 INFO:teuthology.orchestra.run.smithi136.stdout:Adding system user ceph....done 2023-11-16T01:47:16.514 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-base (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:16.641 INFO:teuthology.orchestra.run.smithi136.stdout:Setting system user ceph properties....done 2023-11-16T01:47:16.659 INFO:teuthology.orchestra.run.smithi136.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-16T01:47:16.808 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-16T01:47:16.895 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-16T01:47:17.123 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-16T01:47:17.332 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mds (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:17.559 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-16T01:47:17.559 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-16T01:47:17.578 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-test (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:17.721 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up radosgw (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:17.994 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-16T01:47:17.994 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-16T01:47:18.026 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:18.289 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-16T01:47:18.290 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-16T01:47:18.656 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-base (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:18.743 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-osd (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:19.066 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-16T01:47:19.240 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-16T01:47:19.240 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-16T01:47:19.500 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mds (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:19.662 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr-k8sevents (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:19.731 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-16T01:47:19.731 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-16T01:47:19.788 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr-diskprediction-local (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:19.956 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mon (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:20.168 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:20.180 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-16T01:47:20.180 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-16T01:47:20.392 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-16T01:47:20.393 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-16T01:47:20.624 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr-cephadm (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:20.766 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:20.845 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-osd (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:20.918 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr-dashboard (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:21.061 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-11-16T01:47:21.332 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-16T01:47:21.332 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-16T01:47:21.353 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-11-16T01:47:21.779 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-k8sevents (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:21.905 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-diskprediction-local (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:22.040 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mon (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:22.271 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-16T01:47:22.272 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-16T01:47:22.775 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-cephadm (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:23.070 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:23.218 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-dashboard (16.2.14-171-ged3c8e92-1focal) ... 2023-11-16T01:47:23.362 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-11-16T01:47:23.662 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-11-16T01:47:24.124 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-11-16T01:47:24.562 INFO:teuthology.orchestra.run.smithi083.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-16T01:47:24.565 DEBUG:teuthology.parallel:result is None 2023-11-16T01:47:26.582 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-11-16T01:47:27.022 INFO:teuthology.orchestra.run.smithi136.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-16T01:47:27.024 DEBUG:teuthology.parallel:result is None 2023-11-16T01:47:27.025 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-16T01:47:27.390 DEBUG:teuthology.orchestra.run.smithi083:> dpkg-query -W -f '${Version}' ceph 2023-11-16T01:47:27.412 INFO:teuthology.orchestra.run.smithi083.stdout:16.2.14-171-ged3c8e92-1focal 2023-11-16T01:47:27.413 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-171-ged3c8e92-1focal 2023-11-16T01:47:27.413 INFO:teuthology.task.install:The correct ceph version 16.2.14-171-ged3c8e92-1focal is installed. 2023-11-16T01:47:27.413 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-16T01:47:27.562 DEBUG:teuthology.orchestra.run.smithi136:> dpkg-query -W -f '${Version}' ceph 2023-11-16T01:47:27.584 INFO:teuthology.orchestra.run.smithi136.stdout:16.2.14-171-ged3c8e92-1focal 2023-11-16T01:47:27.584 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-171-ged3c8e92-1focal 2023-11-16T01:47:27.584 INFO:teuthology.task.install:The correct ceph version 16.2.14-171-ged3c8e92-1focal is installed. 2023-11-16T01:47:27.585 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-16T01:47:27.585 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:27.585 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T01:47:27.601 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:47:27.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T01:47:27.619 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-16T01:47:27.620 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:27.620 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T01:47:27.635 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T01:47:27.693 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:47:27.694 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T01:47:27.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T01:47:27.719 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-16T01:47:27.719 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:27.719 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T01:47:27.734 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T01:47:27.794 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:47:27.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T01:47:27.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T01:47:27.865 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-16T01:47:27.866 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:27.866 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T01:47:27.880 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T01:47:27.895 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:47:27.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T01:47:27.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T01:47:27.972 INFO:teuthology.run_tasks:Running task ceph... 2023-11-16T01:47:28.071 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-16T01:47:28.071 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /var/log/ceph 2023-11-16T01:47:28.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /var/log/ceph 2023-11-16T01:47:28.093 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-16T01:47:28.093 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-16T01:47:28.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-16T01:47:28.151 INFO:tasks.ceph:Creating extra log directories... 2023-11-16T01:47:28.153 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-16T01:47:28.155 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-16T01:47:28.182 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-16T01:47:28.182 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-16T01:47:28.182 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694', 'branch': 'wip-yuri3-testing-2023-11-14-1227-pacific', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_workunits}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7459694', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2023-11-14-1227-pacific'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri3-testing-2023-11-14-1227-pacific', 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 7739, 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri3-testing-2023-11-14-1227-pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac', 'targets': {'smithi083.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi136.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 20, 'thrash_delay': 1}}, {'workunit': {'clients': {'client.0': ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-11-15_21:10:46', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1224587'} 2023-11-16T01:47:28.183 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-16T01:47:28.216 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-16T01:47:28.223 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/run/ceph 2023-11-16T01:47:28.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/run/ceph 2023-11-16T01:47:28.239 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:28.240 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2023-11-16T01:47:28.244 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-16T01:47:28.244 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 2023-11-16T01:47:28.293 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-16T01:47:28.293 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T01:47:28.293 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 359 Links: 1 2023-11-16T01:47:28.293 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T01:47:28.293 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2023-11-16 01:43:58.954222194 +0000 2023-11-16T01:47:28.293 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2023-11-16 01:43:58.950222286 +0000 2023-11-16T01:47:28.293 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2023-11-16 01:43:58.950222286 +0000 2023-11-16T01:47:28.293 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2023-11-16T01:47:28.294 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-16T01:47:28.305 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2023-11-16T01:47:28.306 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2023-11-16T01:47:28.306 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000396335 s, 1.3 MB/s 2023-11-16T01:47:28.307 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-16T01:47:28.316 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 2023-11-16T01:47:28.369 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-16T01:47:28.369 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T01:47:28.369 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 367 Links: 1 2023-11-16T01:47:28.369 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T01:47:28.369 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2023-11-16 01:43:59.850201575 +0000 2023-11-16T01:47:28.369 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2023-11-16 01:43:59.478210136 +0000 2023-11-16T01:47:28.369 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2023-11-16 01:43:59.478210136 +0000 2023-11-16T01:47:28.369 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2023-11-16T01:47:28.370 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-16T01:47:28.381 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2023-11-16T01:47:28.381 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2023-11-16T01:47:28.381 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000395452 s, 1.3 MB/s 2023-11-16T01:47:28.382 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-16T01:47:28.391 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 2023-11-16T01:47:28.398 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-16T01:47:28.398 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T01:47:28.398 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 373 Links: 1 2023-11-16T01:47:28.399 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T01:47:28.399 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2023-11-16 01:44:00.406188780 +0000 2023-11-16T01:47:28.399 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2023-11-16 01:44:00.086196144 +0000 2023-11-16T01:47:28.399 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2023-11-16 01:44:00.086196144 +0000 2023-11-16T01:47:28.399 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2023-11-16T01:47:28.399 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-16T01:47:28.415 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2023-11-16T01:47:28.416 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2023-11-16T01:47:28.416 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000399402 s, 1.3 MB/s 2023-11-16T01:47:28.417 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-16T01:47:28.468 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 2023-11-16T01:47:28.478 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-16T01:47:28.478 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T01:47:28.478 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 377 Links: 1 2023-11-16T01:47:28.478 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T01:47:28.478 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2023-11-16 01:44:00.926176813 +0000 2023-11-16T01:47:28.479 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2023-11-16 01:44:00.630183626 +0000 2023-11-16T01:47:28.479 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2023-11-16 01:44:00.630183626 +0000 2023-11-16T01:47:28.479 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2023-11-16T01:47:28.479 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-16T01:47:28.536 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2023-11-16T01:47:28.536 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2023-11-16T01:47:28.536 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000401688 s, 1.3 MB/s 2023-11-16T01:47:28.536 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-16T01:47:28.546 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-16T01:47:28.546 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:47:28.546 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2023-11-16T01:47:28.553 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-16T01:47:28.554 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2023-11-16T01:47:28.563 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-16T01:47:28.564 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T01:47:28.564 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 358 Links: 1 2023-11-16T01:47:28.564 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T01:47:28.564 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-11-16 01:43:59.010627100 +0000 2023-11-16T01:47:28.564 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-11-16 01:43:59.006627191 +0000 2023-11-16T01:47:28.564 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-11-16 01:43:59.006627191 +0000 2023-11-16T01:47:28.564 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-11-16T01:47:28.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-16T01:47:28.574 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-11-16T01:47:28.574 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-11-16T01:47:28.575 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000408441 s, 1.3 MB/s 2023-11-16T01:47:28.575 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-16T01:47:28.586 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2023-11-16T01:47:28.637 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-16T01:47:28.638 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T01:47:28.638 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 365 Links: 1 2023-11-16T01:47:28.638 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T01:47:28.639 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-11-16 01:43:59.890606816 +0000 2023-11-16T01:47:28.639 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-11-16 01:43:59.534615021 +0000 2023-11-16T01:47:28.639 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-11-16 01:43:59.534615021 +0000 2023-11-16T01:47:28.640 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-11-16T01:47:28.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-16T01:47:28.654 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-11-16T01:47:28.654 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-11-16T01:47:28.654 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000441337 s, 1.2 MB/s 2023-11-16T01:47:28.655 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-16T01:47:28.707 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2023-11-16T01:47:28.716 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-16T01:47:28.717 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T01:47:28.717 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 371 Links: 1 2023-11-16T01:47:28.717 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T01:47:28.717 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-11-16 01:44:00.406594921 +0000 2023-11-16T01:47:28.717 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-11-16 01:44:00.126601376 +0000 2023-11-16T01:47:28.717 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-11-16 01:44:00.126601376 +0000 2023-11-16T01:47:28.717 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-11-16T01:47:28.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-16T01:47:28.729 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-11-16T01:47:28.730 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-11-16T01:47:28.730 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000400355 s, 1.3 MB/s 2023-11-16T01:47:28.731 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-16T01:47:28.739 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2023-11-16T01:47:28.745 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-16T01:47:28.745 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T01:47:28.745 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 376 Links: 1 2023-11-16T01:47:28.745 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T01:47:28.746 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-11-16 01:44:00.658589113 +0000 2023-11-16T01:47:28.746 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-11-16 01:44:00.654589205 +0000 2023-11-16T01:47:28.746 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-11-16 01:44:00.654589205 +0000 2023-11-16T01:47:28.746 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-11-16T01:47:28.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-16T01:47:28.801 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-11-16T01:47:28.801 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-11-16T01:47:28.801 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000421147 s, 1.2 MB/s 2023-11-16T01:47:28.802 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-16T01:47:28.809 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-11-16T01:47:28.810 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-11-16T01:47:28.810 INFO:tasks.ceph:Generating config... 2023-11-16T01:47:28.811 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-11-16T01:47:28.811 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-11-16T01:47:28.811 INFO:tasks.ceph:[global] ms inject delay max = 1 2023-11-16T01:47:28.811 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2023-11-16T01:47:28.812 INFO:tasks.ceph:[global] ms inject delay type = mon 2023-11-16T01:47:28.812 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2023-11-16T01:47:28.812 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2023-11-16T01:47:28.812 INFO:tasks.ceph:[global] ms type = async 2023-11-16T01:47:28.812 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-16T01:47:28.812 INFO:tasks.ceph:[mgr] debug monc = 10 2023-11-16T01:47:28.812 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-16T01:47:28.812 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-16T01:47:28.812 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-16T01:47:28.812 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-16T01:47:28.812 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2023-11-16T01:47:28.812 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-11-16T01:47:28.813 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-11-16T01:47:28.813 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-11-16T01:47:28.813 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2023-11-16T01:47:28.813 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2023-11-16T01:47:28.813 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-11-16T01:47:28.813 INFO:tasks.ceph:[mon] paxos service trim min = 5 2023-11-16T01:47:28.813 INFO:tasks.ceph:[osd] bdev async discard = True 2023-11-16T01:47:28.813 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-11-16T01:47:28.813 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-16T01:47:28.813 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-16T01:47:28.813 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-11-16T01:47:28.813 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-11-16T01:47:28.814 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-16T01:47:28.814 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-16T01:47:28.814 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-16T01:47:28.814 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-16T01:47:28.814 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-16T01:47:28.814 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-16T01:47:28.814 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-11-16T01:47:28.814 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-11-16T01:47:28.814 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-16T01:47:28.814 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-11-16T01:47:28.814 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-16T01:47:28.814 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-11-16T01:47:28.815 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-11-16T01:47:28.815 INFO:tasks.ceph:Setting up mon.a... 2023-11-16T01:47:28.815 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-16T01:47:28.872 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/ceph.keyring 2023-11-16T01:47:28.876 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-16T01:47:28.911 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-16T01:47:28.926 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.83'), ('mon.c', '[v2:172.21.15.83:3301,v1:172.21.15.83:6790]'), ('mon.b', '172.21.15.136')] 2023-11-16T01:47:28.927 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.83,[v2:172.21.15.83:3301,v1:172.21.15.83:6790],172.21.15.136', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-11-16T01:47:28.927 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:28.928 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-16T01:47:28.932 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.83 --addv c '[v2:172.21.15.83:3301,v1:172.21.15.83:6790]' --add b 172.21.15.136 --print /home/ubuntu/cephtest/ceph.monmap 2023-11-16T01:47:29.011 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-16T01:47:29.012 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: generated fsid 69ad267b-1e34-4180-8ecf-9774499b7de0 2023-11-16T01:47:29.012 INFO:teuthology.orchestra.run.smithi083.stdout:epoch 0 2023-11-16T01:47:29.012 INFO:teuthology.orchestra.run.smithi083.stdout:fsid 69ad267b-1e34-4180-8ecf-9774499b7de0 2023-11-16T01:47:29.012 INFO:teuthology.orchestra.run.smithi083.stdout:last_changed 2023-11-16T01:47:29.011826+0000 2023-11-16T01:47:29.013 INFO:teuthology.orchestra.run.smithi083.stdout:created 2023-11-16T01:47:29.011826+0000 2023-11-16T01:47:29.013 INFO:teuthology.orchestra.run.smithi083.stdout:min_mon_release 0 (unknown) 2023-11-16T01:47:29.013 INFO:teuthology.orchestra.run.smithi083.stdout:election_strategy: 1 2023-11-16T01:47:29.013 INFO:teuthology.orchestra.run.smithi083.stdout:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2023-11-16T01:47:29.013 INFO:teuthology.orchestra.run.smithi083.stdout:1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.b 2023-11-16T01:47:29.013 INFO:teuthology.orchestra.run.smithi083.stdout:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2023-11-16T01:47:29.013 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-11-16T01:47:29.014 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-16T01:47:29.019 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 69ad267b-1e34-4180-8ecf-9774499b7de0... 2023-11-16T01:47:29.019 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-16T01:47:29.022 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-16T01:47:29.054 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2023-11-16T01:47:29.055 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2023-11-16T01:47:29.055 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-16T01:47:29.055 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2023-11-16T01:47:29.056 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.056 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2023-11-16T01:47:29.056 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.056 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2023-11-16T01:47:29.056 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.057 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2023-11-16T01:47:29.057 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2023-11-16T01:47:29.057 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.057 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2023-11-16T01:47:29.057 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2023-11-16T01:47:29.057 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.057 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-16T01:47:29.057 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2023-11-16T01:47:29.058 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.058 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2023-11-16T01:47:29.058 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-16T01:47:29.058 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.058 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2023-11-16T01:47:29.058 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.059 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2023-11-16T01:47:29.059 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2023-11-16T01:47:29.059 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2023-11-16T01:47:29.059 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2023-11-16T01:47:29.059 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2023-11-16T01:47:29.059 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2023-11-16T01:47:29.059 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2023-11-16T01:47:29.059 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-16T01:47:29.060 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-16T01:47:29.060 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2023-11-16T01:47:29.060 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.060 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-11-16T01:47:29.060 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.060 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2023-11-16T01:47:29.060 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.060 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2023-11-16T01:47:29.061 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.061 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2023-11-16T01:47:29.061 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2023-11-16T01:47:29.061 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2023-11-16T01:47:29.061 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = "172.21.15.83,[v2:172.21.15.83:3301,v1:172.21.15.83:6790],172.21.15.136" 2023-11-16T01:47:29.061 INFO:teuthology.orchestra.run.smithi083.stdout: mon client directed command retry = 5 2023-11-16T01:47:29.061 INFO:teuthology.orchestra.run.smithi083.stdout: mon election default strategy = 1 2023-11-16T01:47:29.061 INFO:teuthology.orchestra.run.smithi083.stdout: ms inject delay max = 1 2023-11-16T01:47:29.061 INFO:teuthology.orchestra.run.smithi083.stdout: ms inject delay probability = 0.005 2023-11-16T01:47:29.062 INFO:teuthology.orchestra.run.smithi083.stdout: ms inject delay type = mon 2023-11-16T01:47:29.062 INFO:teuthology.orchestra.run.smithi083.stdout: ms inject internal delays = 0.002 2023-11-16T01:47:29.062 INFO:teuthology.orchestra.run.smithi083.stdout: ms inject socket failures = 2500 2023-11-16T01:47:29.062 INFO:teuthology.orchestra.run.smithi083.stdout: ms type = async 2023-11-16T01:47:29.062 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 69ad267b-1e34-4180-8ecf-9774499b7de0 2023-11-16T01:47:29.063 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.063 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2023-11-16T01:47:29.063 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2023-11-16T01:47:29.063 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.063 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2023-11-16T01:47:29.063 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2023-11-16T01:47:29.063 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.063 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2023-11-16T01:47:29.063 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2023-11-16T01:47:29.063 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.063 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2023-11-16T01:47:29.063 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2023-11-16T01:47:29.064 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2023-11-16T01:47:29.064 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.064 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2023-11-16T01:47:29.064 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2023-11-16T01:47:29.064 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.064 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2023-11-16T01:47:29.064 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.064 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2023-11-16T01:47:29.064 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.064 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2023-11-16T01:47:29.064 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.065 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2023-11-16T01:47:29.065 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2023-11-16T01:47:29.065 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.065 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2023-11-16T01:47:29.065 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2023-11-16T01:47:29.065 INFO:teuthology.orchestra.run.smithi083.stdout: bdev async discard = True 2023-11-16T01:47:29.065 INFO:teuthology.orchestra.run.smithi083.stdout: bdev enable discard = True 2023-11-16T01:47:29.065 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore block size = 96636764160 2023-11-16T01:47:29.065 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore fsck on mount = True 2023-11-16T01:47:29.065 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluefs = 20 2023-11-16T01:47:29.065 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluestore = 20 2023-11-16T01:47:29.065 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2023-11-16T01:47:29.066 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2023-11-16T01:47:29.066 INFO:teuthology.orchestra.run.smithi083.stdout: debug rocksdb = 10 2023-11-16T01:47:29.066 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd backfillfull_ratio = 0.85 2023-11-16T01:47:29.067 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd full ratio = 0.9 2023-11-16T01:47:29.067 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd nearfull ratio = 0.8 2023-11-16T01:47:29.067 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify cached snaps = True 2023-11-16T01:47:29.067 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify missing on start = True 2023-11-16T01:47:29.067 INFO:teuthology.orchestra.run.smithi083.stdout: osd failsafe full ratio = 0.95 2023-11-16T01:47:29.067 INFO:teuthology.orchestra.run.smithi083.stdout: osd memory target = 1610612736 2023-11-16T01:47:29.067 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = bluestore 2023-11-16T01:47:29.067 INFO:teuthology.orchestra.run.smithi083.stdout: osd op queue = debug_random 2023-11-16T01:47:29.067 INFO:teuthology.orchestra.run.smithi083.stdout: osd op queue cut off = debug_random 2023-11-16T01:47:29.067 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.067 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2023-11-16T01:47:29.068 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2023-11-16T01:47:29.068 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2023-11-16T01:47:29.068 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2023-11-16T01:47:29.068 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2023-11-16T01:47:29.068 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2023-11-16T01:47:29.068 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2023-11-16T01:47:29.068 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2023-11-16T01:47:29.068 INFO:teuthology.orchestra.run.smithi083.stdout: debug monc = 10 2023-11-16T01:47:29.068 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.068 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2023-11-16T01:47:29.069 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2023-11-16T01:47:29.069 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2023-11-16T01:47:29.069 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2023-11-16T01:47:29.069 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2023-11-16T01:47:29.069 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2023-11-16T01:47:29.069 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2023-11-16T01:47:29.069 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2023-11-16T01:47:29.069 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2023-11-16T01:47:29.069 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2023-11-16T01:47:29.069 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2023-11-16T01:47:29.070 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.070 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-16T01:47:29.070 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2023-11-16T01:47:29.070 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2023-11-16T01:47:29.070 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.070 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2023-11-16T01:47:29.070 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-16T01:47:29.070 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-16T01:47:29.070 INFO:teuthology.orchestra.run.smithi083.stdout: mon min osdmap epochs = 25 2023-11-16T01:47:29.070 INFO:teuthology.orchestra.run.smithi083.stdout: mon osdmap full prune interval = 2 2023-11-16T01:47:29.070 INFO:teuthology.orchestra.run.smithi083.stdout: mon osdmap full prune min = 15 2023-11-16T01:47:29.071 INFO:teuthology.orchestra.run.smithi083.stdout: mon osdmap full prune txsize = 2 2023-11-16T01:47:29.071 INFO:teuthology.orchestra.run.smithi083.stdout: mon scrub inject crc mismatch = 0.01 2023-11-16T01:47:29.071 INFO:teuthology.orchestra.run.smithi083.stdout: mon scrub inject missing keys = 0.05 2023-11-16T01:47:29.071 INFO:teuthology.orchestra.run.smithi083.stdout: mon scrub interval = 300 2023-11-16T01:47:29.071 INFO:teuthology.orchestra.run.smithi083.stdout: paxos service trim min = 5 2023-11-16T01:47:29.071 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:47:29.071 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2023-11-16T01:47:29.071 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2023-11-16T01:47:29.071 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2023-11-16T01:47:29.071 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2023-11-16T01:47:29.072 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-16T01:47:29.072 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-16T01:47:29.072 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2023-11-16T01:47:29.072 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.c] 2023-11-16T01:47:29.072 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.b] 2023-11-16T01:47:29.073 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2023-11-16T01:47:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: chdir = "" 2023-11-16T01:47:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-16T01:47:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: auth supported = cephx 2023-11-16T01:47:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: filestore xattr use omap = true 2023-11-16T01:47:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: mon clock drift allowed = 1.000 2023-11-16T01:47:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: osd crush chooseleaf type = 0 2023-11-16T01:47:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: auth debug = true 2023-11-16T01:47:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on old message = true 2023-11-16T01:47:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on bug = true 2023-11-16T01:47:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-16T01:47:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: mon pg warn max object skew = 0 2023-11-16T01:47:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: # disable pg_autoscaler by default for new pools 2023-11-16T01:47:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-16T01:47:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default size = 2 2023-11-16T01:47:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow primary affinity = true 2023-11-16T01:47:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow pg remap = true 2023-11-16T01:47:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on legacy crush tunables = false 2023-11-16T01:47:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on crush straw calc version zero = false 2023-11-16T01:47:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on no sortbitwise = false 2023-11-16T01:47:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on osd down out interval zero = false 2023-11-16T01:47:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on too few osds = false 2023-11-16T01:47:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-16T01:47:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-16T01:47:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: mon_allow_pool_size_one = true 2023-11-16T01:47:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-11-16T01:47:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: osd default data pool replay window = 5 2023-11-16T01:47:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: mon allow pool delete = true 2023-11-16T01:47:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: mon cluster log file level = debug 2023-11-16T01:47:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: debug asserts on shutdown = true 2023-11-16T01:47:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: mon health detail to clog = false 2023-11-16T01:47:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: mon host = "172.21.15.83,[v2:172.21.15.83:3301,v1:172.21.15.83:6790],172.21.15.136" 2023-11-16T01:47:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: mon client directed command retry = 5 2023-11-16T01:47:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: mon election default strategy = 1 2023-11-16T01:47:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject delay max = 1 2023-11-16T01:47:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject delay probability = 0.005 2023-11-16T01:47:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject delay type = mon 2023-11-16T01:47:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject internal delays = 0.002 2023-11-16T01:47:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject socket failures = 2500 2023-11-16T01:47:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: ms type = async 2023-11-16T01:47:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: fsid = 69ad267b-1e34-4180-8ecf-9774499b7de0 2023-11-16T01:47:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.078 INFO:teuthology.orchestra.run.smithi136.stdout:[osd] 2023-11-16T01:47:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: osd journal size = 100 2023-11-16T01:47:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub load threshold = 5.0 2023-11-16T01:47:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub max interval = 600 2023-11-16T01:47:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: osd recover clone overlap = true 2023-11-16T01:47:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: osd recovery max chunk = 1048576 2023-11-16T01:47:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug shutdown = true 2023-11-16T01:47:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug op order = true 2023-11-16T01:47:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify stray on activate = true 2023-11-16T01:47:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: osd open classes on start = true 2023-11-16T01:47:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug pg log writeout = true 2023-11-16T01:47:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: osd deep scrub update digest min age = 30 2023-11-16T01:47:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: osd map max advance = 10 2023-11-16T01:47:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: journal zero on create = true 2023-11-16T01:47:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: filestore ondisk finisher threads = 3 2023-11-16T01:47:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: filestore apply finisher threads = 3 2023-11-16T01:47:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: bdev debug aio = true 2023-11-16T01:47:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug misdirected ops = true 2023-11-16T01:47:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: bdev async discard = True 2023-11-16T01:47:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: bdev enable discard = True 2023-11-16T01:47:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore block size = 96636764160 2023-11-16T01:47:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore fsck on mount = True 2023-11-16T01:47:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluefs = 20 2023-11-16T01:47:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluestore = 20 2023-11-16T01:47:29.082 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2023-11-16T01:47:29.082 INFO:teuthology.orchestra.run.smithi136.stdout: debug osd = 20 2023-11-16T01:47:29.082 INFO:teuthology.orchestra.run.smithi136.stdout: debug rocksdb = 10 2023-11-16T01:47:29.082 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd backfillfull_ratio = 0.85 2023-11-16T01:47:29.082 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd full ratio = 0.9 2023-11-16T01:47:29.082 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd nearfull ratio = 0.8 2023-11-16T01:47:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify cached snaps = True 2023-11-16T01:47:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify missing on start = True 2023-11-16T01:47:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: osd failsafe full ratio = 0.95 2023-11-16T01:47:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: osd memory target = 1610612736 2023-11-16T01:47:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: osd objectstore = bluestore 2023-11-16T01:47:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: osd op queue = debug_random 2023-11-16T01:47:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: osd op queue cut off = debug_random 2023-11-16T01:47:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.083 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr] 2023-11-16T01:47:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2023-11-16T01:47:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: debug mgr = 20 2023-11-16T01:47:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2023-11-16T01:47:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2023-11-16T01:47:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2023-11-16T01:47:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2023-11-16T01:47:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: mgr/telemetry/nag = false 2023-11-16T01:47:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: debug monc = 10 2023-11-16T01:47:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.085 INFO:teuthology.orchestra.run.smithi136.stdout:[mon] 2023-11-16T01:47:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2023-11-16T01:47:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2023-11-16T01:47:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: debug paxos = 20 2023-11-16T01:47:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2023-11-16T01:47:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: mon data avail warn = 5 2023-11-16T01:47:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: mon mgr mkfs grace = 240 2023-11-16T01:47:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2023-11-16T01:47:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd reporter subtree level = osd 2023-11-16T01:47:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd prime pg temp = true 2023-11-16T01:47:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2023-11-16T01:47:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-16T01:47:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: auth mon ticket ttl = 660 # 11m 2023-11-16T01:47:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: auth service ticket ttl = 240 # 4m 2023-11-16T01:47:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: # don't complain about insecure global_id in the test suite 2023-11-16T01:47:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-16T01:47:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-16T01:47:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: mon min osdmap epochs = 25 2023-11-16T01:47:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: mon osdmap full prune interval = 2 2023-11-16T01:47:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: mon osdmap full prune min = 15 2023-11-16T01:47:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: mon osdmap full prune txsize = 2 2023-11-16T01:47:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: mon scrub inject crc mismatch = 0.01 2023-11-16T01:47:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: mon scrub inject missing keys = 0.05 2023-11-16T01:47:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: mon scrub interval = 300 2023-11-16T01:47:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: paxos service trim min = 5 2023-11-16T01:47:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-11-16T01:47:29.087 INFO:teuthology.orchestra.run.smithi136.stdout:[client] 2023-11-16T01:47:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: rgw cache enabled = true 2023-11-16T01:47:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable ops log = true 2023-11-16T01:47:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable usage log = true 2023-11-16T01:47:29.088 INFO:teuthology.orchestra.run.smithi136.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-16T01:47:29.088 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-16T01:47:29.088 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.a] 2023-11-16T01:47:29.088 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.c] 2023-11-16T01:47:29.088 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.b] 2023-11-16T01:47:29.088 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-16T01:47:29.089 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-16T01:47:29.136 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-16T01:47:29.136 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:29.136 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-16T01:47:29.186 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:29.186 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-16T01:47:29.198 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi083.front.sepia.ceph.com 2023-11-16T01:47:29.199 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:29.199 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-16T01:47:29.200 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-16T01:47:29.263 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:29.263 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-16T01:47:29.270 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi136.front.sepia.ceph.com 2023-11-16T01:47:29.270 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:47:29.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-16T01:47:29.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-16T01:47:29.299 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:47:29.300 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-16T01:47:29.348 INFO:tasks.ceph:Setting up mon nodes... 2023-11-16T01:47:29.349 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-16T01:47:29.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-16T01:47:29.405 INFO:teuthology.orchestra.run.smithi136.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-16T01:47:29.408 INFO:tasks.ceph:Setting up mds nodes... 2023-11-16T01:47:29.408 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-16T01:47:29.408 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-16T01:47:29.442 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-16T01:47:29.455 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-16T01:47:29.455 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-11-16T01:47:29.455 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-16T01:47:29.472 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-16T01:47:29.472 INFO:tasks.ceph:role: osd.0 2023-11-16T01:47:29.472 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com 2023-11-16T01:47:29.472 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-16T01:47:30.024 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-16T01:47:30.024 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-16T01:47:30.024 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-16T01:47:30.024 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 2023-11-16T01:47:30.024 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-16T01:47:30.024 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2023-11-16T01:47:30.024 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-16T01:47:30.024 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-16T01:47:30.024 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-16T01:47:30.024 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-16T01:47:30.026 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2023-11-16T01:47:30.026 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-16T01:47:30.121 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-16T01:47:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: /sbin/restorecon: command not found 2023-11-16T01:47:30.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:47:30.137 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-16T01:47:30.196 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-16T01:47:30.197 INFO:tasks.ceph:role: osd.1 2023-11-16T01:47:30.197 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com 2023-11-16T01:47:30.197 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-16T01:47:30.703 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-16T01:47:30.703 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-16T01:47:30.704 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-16T01:47:30.704 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 2023-11-16T01:47:30.704 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-16T01:47:30.704 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2023-11-16T01:47:30.704 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-16T01:47:30.705 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-16T01:47:30.705 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-16T01:47:30.705 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-16T01:47:30.706 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2023-11-16T01:47:30.707 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-16T01:47:30.745 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-16T01:47:30.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:47:30.757 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: /sbin/restorecon: command not found 2023-11-16T01:47:30.757 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-16T01:47:30.815 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-16T01:47:30.815 INFO:tasks.ceph:role: osd.2 2023-11-16T01:47:30.817 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com 2023-11-16T01:47:30.817 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-16T01:47:31.285 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-16T01:47:31.285 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-16T01:47:31.285 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-16T01:47:31.285 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 2023-11-16T01:47:31.285 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-16T01:47:31.286 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2023-11-16T01:47:31.286 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-16T01:47:31.286 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-16T01:47:31.286 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-16T01:47:31.286 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-16T01:47:31.287 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2023-11-16T01:47:31.288 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-16T01:47:31.342 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-16T01:47:31.354 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: /sbin/restorecon: command not found 2023-11-16T01:47:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:47:31.355 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-16T01:47:31.471 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:31.468+0000 7f2f96f3f200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-11-16T01:47:31.472 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:31.468+0000 7f2f96f3f200 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-11-16T01:47:31.472 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:31.468+0000 7f2f96f3f200 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-16T01:47:31.472 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:31.468+0000 7f2f96f3f200 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-16T01:47:31.473 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:31.468+0000 7f2f96f3f200 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-16T01:47:31.473 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:31.468+0000 7f2f96f3f200 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-11-16T01:47:34.478 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-16T01:47:34.496 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-16T01:47:34.580 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:34.576+0000 7f62d6930200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-11-16T01:47:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:34.576+0000 7f62d6930200 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-11-16T01:47:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:34.576+0000 7f62d6930200 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-16T01:47:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:34.576+0000 7f62d6930200 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-16T01:47:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:34.576+0000 7f62d6930200 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-16T01:47:34.582 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:34.576+0000 7f62d6930200 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-11-16T01:47:37.578 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-16T01:47:37.594 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-16T01:47:37.629 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:37.624+0000 7f2397e71200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-11-16T01:47:37.629 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:37.624+0000 7f2397e71200 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-11-16T01:47:37.630 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:37.624+0000 7f2397e71200 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-16T01:47:37.630 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:37.624+0000 7f2397e71200 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-16T01:47:37.630 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:37.624+0000 7f2397e71200 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-16T01:47:37.630 INFO:teuthology.orchestra.run.smithi083.stderr:2023-11-16T01:47:37.628+0000 7f2397e71200 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-11-16T01:47:40.621 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-16T01:47:40.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-11-16T01:47:40.694 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-11-16T01:47:40.694 INFO:tasks.ceph:role: osd.3 2023-11-16T01:47:40.694 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com 2023-11-16T01:47:40.695 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-16T01:47:41.301 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-16T01:47:41.301 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-16T01:47:41.301 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-16T01:47:41.301 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 2023-11-16T01:47:41.301 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-16T01:47:41.301 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2023-11-16T01:47:41.301 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-16T01:47:41.302 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-16T01:47:41.302 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-16T01:47:41.302 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-16T01:47:41.303 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2023-11-16T01:47:41.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2023-11-16T01:47:41.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-11-16T01:47:41.376 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2023-11-16T01:47:41.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:47:41.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-11-16T01:47:41.387 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-11-16T01:47:41.388 INFO:tasks.ceph:role: osd.4 2023-11-16T01:47:41.388 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com 2023-11-16T01:47:41.388 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-16T01:47:41.889 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-16T01:47:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-16T01:47:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-16T01:47:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 2023-11-16T01:47:41.889 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-16T01:47:41.890 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2023-11-16T01:47:41.890 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-16T01:47:41.890 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-16T01:47:41.890 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-16T01:47:41.890 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-16T01:47:41.891 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2023-11-16T01:47:41.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2023-11-16T01:47:41.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-11-16T01:47:41.926 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2023-11-16T01:47:41.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:47:41.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-11-16T01:47:41.938 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-11-16T01:47:41.938 INFO:tasks.ceph:role: osd.5 2023-11-16T01:47:41.938 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com 2023-11-16T01:47:41.938 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-16T01:47:42.685 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-16T01:47:42.685 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-16T01:47:42.685 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-16T01:47:42.685 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 2023-11-16T01:47:42.685 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-16T01:47:42.685 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2023-11-16T01:47:42.685 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-16T01:47:42.685 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-16T01:47:42.685 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-16T01:47:42.686 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-16T01:47:42.686 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2023-11-16T01:47:42.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2023-11-16T01:47:42.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-11-16T01:47:42.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:47:42.773 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2023-11-16T01:47:42.773 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-16T01:47:42.852 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:42.850+0000 7f8b227bd200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-11-16T01:47:42.852 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:42.850+0000 7f8b227bd200 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-11-16T01:47:42.852 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:42.850+0000 7f8b227bd200 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-16T01:47:42.852 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:42.850+0000 7f8b227bd200 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-16T01:47:42.852 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:42.850+0000 7f8b227bd200 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-16T01:47:42.856 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:42.854+0000 7f8b227bd200 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-11-16T01:47:45.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-11-16T01:47:45.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-16T01:47:45.940 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:45.938+0000 7fcb10498200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-11-16T01:47:45.941 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:45.938+0000 7fcb10498200 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-11-16T01:47:45.941 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:45.938+0000 7fcb10498200 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-16T01:47:45.941 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:45.938+0000 7fcb10498200 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-16T01:47:45.941 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:45.938+0000 7fcb10498200 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-16T01:47:45.942 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:45.938+0000 7fcb10498200 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-11-16T01:47:48.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-11-16T01:47:48.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-16T01:47:48.978 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:48.974+0000 7fbbd3d68200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-11-16T01:47:48.978 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:48.974+0000 7fbbd3d68200 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-11-16T01:47:48.978 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:48.974+0000 7fbbd3d68200 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-16T01:47:48.978 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:48.974+0000 7fbbd3d68200 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-16T01:47:48.978 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:48.974+0000 7fbbd3d68200 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-16T01:47:48.979 INFO:teuthology.orchestra.run.smithi136.stderr:2023-11-16T01:47:48.978+0000 7fbbd3d68200 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-11-16T01:47:51.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-11-16T01:47:52.022 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-16T01:47:52.022 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:52.022 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-16T01:47:52.047 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:52.047 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-16T01:47:52.106 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:52.107 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-16T01:47:52.117 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:47:52.118 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-16T01:47:52.138 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:47:52.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-11-16T01:47:52.198 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:47:52.199 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-11-16T01:47:52.210 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:47:52.211 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-11-16T01:47:52.275 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:47:52.275 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-16T01:47:52.325 INFO:tasks.ceph:Adding keys to all mons... 2023-11-16T01:47:52.325 DEBUG:teuthology.orchestra.run.smithi083:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-16T01:47:52.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-16T01:47:52.342 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.0] 2023-11-16T01:47:52.342 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCzdFVlPnkZHBAA4oROvScle4r+d77LoH5nYg== 2023-11-16T01:47:52.342 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.1] 2023-11-16T01:47:52.342 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQC2dFVlhWqVIhAACXIA0LRVXltv20i/Nr8SyQ== 2023-11-16T01:47:52.342 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.2] 2023-11-16T01:47:52.343 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQC5dFVlOROGJRAAwpDpr/DhVD7kGnX7CiWJmQ== 2023-11-16T01:47:52.343 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr.x] 2023-11-16T01:47:52.344 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCxdFVltmcvGBAAV182sWm7dsoyIbyTO3Xtzw== 2023-11-16T01:47:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.3] 2023-11-16T01:47:52.344 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQC+dFVlB+HQMhAAuD9CZvaKsBkFuEXgg47NOg== 2023-11-16T01:47:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.4] 2023-11-16T01:47:52.344 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQDBdFVl+Z0dOBAA4YZcC/H8XX3PYEXhM9r3wA== 2023-11-16T01:47:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.5] 2023-11-16T01:47:52.345 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQDEdFVlqZpQOhAAc2Rg04WhhxxueVew1EwfVw== 2023-11-16T01:47:52.345 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2023-11-16T01:47:52.345 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCxdFVlqmtvGhAAT8FVBK/Na28pW5ZiAdS8AA== 2023-11-16T01:47:52.353 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.0] 2023-11-16T01:47:52.353 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCzdFVlPnkZHBAA4oROvScle4r+d77LoH5nYg== 2023-11-16T01:47:52.353 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.1] 2023-11-16T01:47:52.353 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQC2dFVlhWqVIhAACXIA0LRVXltv20i/Nr8SyQ== 2023-11-16T01:47:52.353 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.2] 2023-11-16T01:47:52.354 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQC5dFVlOROGJRAAwpDpr/DhVD7kGnX7CiWJmQ== 2023-11-16T01:47:52.354 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.x] 2023-11-16T01:47:52.354 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCxdFVltmcvGBAAV182sWm7dsoyIbyTO3Xtzw== 2023-11-16T01:47:52.354 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.3] 2023-11-16T01:47:52.354 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQC+dFVlB+HQMhAAuD9CZvaKsBkFuEXgg47NOg== 2023-11-16T01:47:52.354 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.4] 2023-11-16T01:47:52.354 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDBdFVl+Z0dOBAA4YZcC/H8XX3PYEXhM9r3wA== 2023-11-16T01:47:52.354 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.5] 2023-11-16T01:47:52.354 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDEdFVlqZpQOhAAc2Rg04WhhxxueVew1EwfVw== 2023-11-16T01:47:52.354 INFO:teuthology.orchestra.run.smithi136.stdout:[client.0] 2023-11-16T01:47:52.355 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCxdFVlqmtvGhAAT8FVBK/Na28pW5ZiAdS8AA== 2023-11-16T01:47:52.355 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-16T01:47:52.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-16T01:47:52.455 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-16T01:47:52.461 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-16T01:47:52.507 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-16T01:47:52.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-16T01:47:52.595 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-16T01:47:52.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-16T01:47:52.645 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-16T01:47:52.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-16T01:47:52.732 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-16T01:47:52.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-16T01:47:52.779 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-16T01:47:52.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-16T01:47:52.829 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-16T01:47:52.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-16T01:47:52.916 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-16T01:47:52.916 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-16T01:47:52.929 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-16T01:47:52.979 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-16T01:47:52.993 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-11-16T01:47:53.006 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-16T01:47:53.057 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-11-16T01:47:53.069 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-11-16T01:47:53.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-16T01:47:53.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-11-16T01:47:53.197 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-16T01:47:53.202 DEBUG:teuthology.orchestra.run.smithi136:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-16T01:47:53.207 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-16T01:47:53.207 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-16T01:47:53.208 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-16T01:47:53.251 INFO:tasks.ceph.mon.a:Started 2023-11-16T01:47:53.252 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-16T01:47:53.252 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-11-16T01:47:53.254 INFO:tasks.ceph.mon.c:Started 2023-11-16T01:47:53.254 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-16T01:47:53.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-16T01:47:53.256 INFO:tasks.ceph.mon.b:Started 2023-11-16T01:47:53.257 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-16T01:47:53.257 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-16T01:47:53.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-16T01:47:53.259 INFO:tasks.ceph.mgr.x:Started 2023-11-16T01:47:53.260 INFO:tasks.ceph:Setting crush tunables to default 2023-11-16T01:47:53.260 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd crush tunables default 2023-11-16T01:47:59.080 INFO:teuthology.orchestra.run.smithi083.stderr:adjusted tunables profile to default 2023-11-16T01:47:59.090 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-16T01:47:59.091 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:59.091 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-16T01:47:59.105 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:59.105 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-16T01:47:59.159 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2023-11-16T01:47:59.159 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-16T01:47:59.169 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:47:59.165+0000 7f6424825500 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T01:47:59.175 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:47:59.175 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-11-16T01:47:59.191 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:47:59.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-11-16T01:47:59.229 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:47:59.225+0000 7f6424825500 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T01:47:59.243 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:47:59.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-11-16T01:47:59.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 9244fa2e-56f6-41bf-b5e8-264183343a6d 0 2023-11-16T01:48:00.194 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:00.189+0000 7f6424825500 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T01:48:00.347 INFO:teuthology.orchestra.run.smithi136.stdout:0 2023-11-16T01:48:00.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new ed5dde4d-7737-428e-af42-a56e6224529d 1 2023-11-16T01:48:00.584 INFO:teuthology.orchestra.run.smithi136.stdout:1 2023-11-16T01:48:00.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new b363e0df-3ed6-4fd7-b06a-6fc5f880f606 2 2023-11-16T01:48:00.797 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:00.793+0000 7f6424825500 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T01:48:00.953 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:00.949+0000 7f6424825500 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T01:48:01.174 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:01.169+0000 7f6424825500 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T01:48:01.326 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:01.321+0000 7f6424825500 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T01:48:01.374 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:01.369+0000 7f6424825500 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T01:48:01.435 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:01.433+0000 7f6424825500 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T01:48:01.488 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:01.485+0000 7f6424825500 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T01:48:01.628 INFO:teuthology.orchestra.run.smithi136.stdout:2 2023-11-16T01:48:01.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new a24e3b64-a290-479e-ab17-c35a891960c2 3 2023-11-16T01:48:01.789 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:01.785+0000 7f6424825500 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T01:48:01.858 INFO:teuthology.orchestra.run.smithi136.stdout:3 2023-11-16T01:48:01.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new f3f49f6a-7a3d-4985-a8c9-995777f7fda5 4 2023-11-16T01:48:01.946 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:01.941+0000 7f6424825500 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T01:48:02.072 INFO:teuthology.orchestra.run.smithi136.stdout:4 2023-11-16T01:48:02.092 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new ee8f2910-b899-4798-a9d8-762ea0e7c9b6 5 2023-11-16T01:48:02.205 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:02.201+0000 7f6424825500 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T01:48:02.274 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:02.269+0000 7f6424825500 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T01:48:02.522 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:02.517+0000 7f6424825500 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T01:48:02.576 INFO:teuthology.orchestra.run.smithi136.stdout:5 2023-11-16T01:48:02.583 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:02.577+0000 7f6424825500 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T01:48:02.593 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-16T01:48:02.594 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-11-16T01:48:02.596 INFO:tasks.ceph.osd.0:Started 2023-11-16T01:48:02.597 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-16T01:48:02.597 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-11-16T01:48:02.599 INFO:tasks.ceph.osd.1:Started 2023-11-16T01:48:02.599 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-16T01:48:02.599 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-11-16T01:48:02.601 INFO:tasks.ceph.osd.2:Started 2023-11-16T01:48:02.602 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-16T01:48:02.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-11-16T01:48:02.603 INFO:tasks.ceph.osd.3:Started 2023-11-16T01:48:02.604 INFO:tasks.ceph.osd.4:Restarting daemon 2023-11-16T01:48:02.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-11-16T01:48:02.606 INFO:tasks.ceph.osd.4:Started 2023-11-16T01:48:02.606 INFO:tasks.ceph.osd.5:Restarting daemon 2023-11-16T01:48:02.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-11-16T01:48:02.612 INFO:tasks.ceph.osd.5:Started 2023-11-16T01:48:02.612 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T01:48:02.754 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:02.749+0000 7f6424825500 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T01:48:02.808 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:48:02.808 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":8,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","created":"2023-11-16T01:47:58.606709+0000","modified":"2023-11-16T01:48:02.279946+0000","last_up_change":"0.000000","last_in_change":"2023-11-16T01:48:02.279946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"9244fa2e-56f6-41bf-b5e8-264183343a6d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ed5dde4d-7737-428e-af42-a56e6224529d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b363e0df-3ed6-4fd7-b06a-6fc5f880f606","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a24e3b64-a290-479e-ab17-c35a891960c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f3f49f6a-7a3d-4985-a8c9-995777f7fda5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee8f2910-b899-4798-a9d8-762ea0e7c9b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T01:48:02.820 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-16T01:48:02.821 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-16T01:48:02.850 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:02.845+0000 7f6424825500 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T01:48:02.990 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:02.985+0000 7f6424825500 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T01:48:03.013 INFO:tasks.ceph.osd.0.smithi083.stderr:2023-11-16T01:48:03.003+0000 7f8a7b5ff200 -1 Falling back to public interface 2023-11-16T01:48:03.181 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:03.177+0000 7f6424825500 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T01:48:03.216 INFO:tasks.ceph.osd.1.smithi083.stderr:2023-11-16T01:48:03.211+0000 7f6f7a6d0200 -1 Falling back to public interface 2023-11-16T01:48:03.216 INFO:tasks.ceph.osd.2.smithi083.stderr:2023-11-16T01:48:03.211+0000 7f1f9d475200 -1 Falling back to public interface 2023-11-16T01:48:03.227 INFO:tasks.ceph.osd.3.smithi136.stderr:2023-11-16T01:48:03.221+0000 7f587a5f1200 -1 Falling back to public interface 2023-11-16T01:48:03.227 INFO:tasks.ceph.osd.5.smithi136.stderr:2023-11-16T01:48:03.221+0000 7f55f1591200 -1 Falling back to public interface 2023-11-16T01:48:03.252 INFO:tasks.ceph.osd.4.smithi136.stderr:2023-11-16T01:48:03.249+0000 7f6fb497d200 -1 Falling back to public interface 2023-11-16T01:48:03.432 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-16T01:48:03.610 INFO:teuthology.misc.health.smithi083.stdout: 2023-11-16T01:48:03.611 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":8,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","created":"2023-11-16T01:47:58.606709+0000","modified":"2023-11-16T01:48:02.279946+0000","last_up_change":"0.000000","last_in_change":"2023-11-16T01:48:02.279946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"9244fa2e-56f6-41bf-b5e8-264183343a6d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ed5dde4d-7737-428e-af42-a56e6224529d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b363e0df-3ed6-4fd7-b06a-6fc5f880f606","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a24e3b64-a290-479e-ab17-c35a891960c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f3f49f6a-7a3d-4985-a8c9-995777f7fda5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee8f2910-b899-4798-a9d8-762ea0e7c9b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T01:48:03.622 DEBUG:teuthology.misc:0 of 6 OSDs are up 2023-11-16T01:48:06.262 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:06.257+0000 7f640e52e700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-16T01:48:06.262 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-11-16T01:48:06.257+0000 7f640e52e700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-16T01:48:06.763 INFO:tasks.ceph.osd.0.smithi083.stderr:2023-11-16T01:48:06.759+0000 7f8a7b5ff200 -1 osd.0 0 log_to_monitors {default=true} 2023-11-16T01:48:06.855 INFO:tasks.ceph.osd.1.smithi083.stderr:2023-11-16T01:48:06.851+0000 7f6f7a6d0200 -1 osd.1 0 log_to_monitors {default=true} 2023-11-16T01:48:06.918 INFO:tasks.ceph.osd.3.smithi136.stderr:2023-11-16T01:48:06.913+0000 7f587a5f1200 -1 osd.3 0 log_to_monitors {default=true} 2023-11-16T01:48:06.963 INFO:tasks.ceph.osd.2.smithi083.stderr:2023-11-16T01:48:06.959+0000 7f1f9d475200 -1 osd.2 0 log_to_monitors {default=true} 2023-11-16T01:48:07.019 INFO:tasks.ceph.osd.5.smithi136.stderr:2023-11-16T01:48:07.013+0000 7f55f1591200 -1 osd.5 0 log_to_monitors {default=true} 2023-11-16T01:48:07.064 INFO:tasks.ceph.osd.4.smithi136.stderr:2023-11-16T01:48:07.061+0000 7f6fb497d200 -1 osd.4 0 log_to_monitors {default=true} 2023-11-16T01:48:08.282 INFO:tasks.ceph.osd.0.smithi083.stderr:2023-11-16T01:48:08.275+0000 7f8a76d9f700 -1 osd.0 0 waiting for initial osdmap 2023-11-16T01:48:08.282 INFO:tasks.ceph.osd.1.smithi083.stderr:2023-11-16T01:48:08.279+0000 7f6f75e70700 -1 osd.1 0 waiting for initial osdmap 2023-11-16T01:48:08.283 INFO:tasks.ceph.osd.2.smithi083.stderr:2023-11-16T01:48:08.279+0000 7f1f98c15700 -1 osd.2 0 waiting for initial osdmap 2023-11-16T01:48:08.286 INFO:tasks.ceph.osd.3.smithi136.stderr:2023-11-16T01:48:08.281+0000 7f5875d91700 -1 osd.3 0 waiting for initial osdmap 2023-11-16T01:48:08.286 INFO:tasks.ceph.osd.4.smithi136.stderr:2023-11-16T01:48:08.281+0000 7f6fb011d700 -1 osd.4 0 waiting for initial osdmap 2023-11-16T01:48:08.286 INFO:tasks.ceph.osd.5.smithi136.stderr:2023-11-16T01:48:08.281+0000 7f55ee556700 -1 osd.5 0 waiting for initial osdmap 2023-11-16T01:48:08.289 INFO:tasks.ceph.osd.0.smithi083.stderr:2023-11-16T01:48:08.283+0000 7f8a7130e700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T01:48:08.290 INFO:tasks.ceph.osd.1.smithi083.stderr:2023-11-16T01:48:08.283+0000 7f6f703df700 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T01:48:08.290 INFO:tasks.ceph.osd.2.smithi083.stderr:2023-11-16T01:48:08.287+0000 7f1f93985700 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T01:48:08.295 INFO:tasks.ceph.osd.3.smithi136.stderr:2023-11-16T01:48:08.293+0000 7f5870b01700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T01:48:08.295 INFO:tasks.ceph.osd.4.smithi136.stderr:2023-11-16T01:48:08.293+0000 7f6faa68c700 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T01:48:08.295 INFO:tasks.ceph.osd.5.smithi136.stderr:2023-11-16T01:48:08.293+0000 7f55e7aa1700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T01:48:10.228 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-16T01:48:10.502 INFO:teuthology.misc.health.smithi083.stdout: 2023-11-16T01:48:10.502 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":12,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","created":"2023-11-16T01:47:58.606709+0000","modified":"2023-11-16T01:48:10.292390+0000","last_up_change":"2023-11-16T01:48:09.279683+0000","last_in_change":"2023-11-16T01:48:02.279946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-16T01:48:09.347564+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"9244fa2e-56f6-41bf-b5e8-264183343a6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":30853},{"type":"v1","addr":"172.21.15.83:6801","nonce":30853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":30853},{"type":"v1","addr":"172.21.15.83:6803","nonce":30853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":30853},{"type":"v1","addr":"172.21.15.83:6807","nonce":30853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":30853},{"type":"v1","addr":"172.21.15.83:6805","nonce":30853}]},"public_addr":"172.21.15.83:6801/30853","cluster_addr":"172.21.15.83:6803/30853","heartbeat_back_addr":"172.21.15.83:6807/30853","heartbeat_front_addr":"172.21.15.83:6805/30853","state":["exists","up"]},{"osd":1,"uuid":"ed5dde4d-7737-428e-af42-a56e6224529d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":30851},{"type":"v1","addr":"172.21.15.83:6809","nonce":30851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6811","nonce":30851},{"type":"v1","addr":"172.21.15.83:6813","nonce":30851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6819","nonce":30851},{"type":"v1","addr":"172.21.15.83:6821","nonce":30851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":30851},{"type":"v1","addr":"172.21.15.83:6817","nonce":30851}]},"public_addr":"172.21.15.83:6809/30851","cluster_addr":"172.21.15.83:6813/30851","heartbeat_back_addr":"172.21.15.83:6821/30851","heartbeat_front_addr":"172.21.15.83:6817/30851","state":["exists","up"]},{"osd":2,"uuid":"b363e0df-3ed6-4fd7-b06a-6fc5f880f606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":30852},{"type":"v1","addr":"172.21.15.83:6812","nonce":30852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":30852},{"type":"v1","addr":"172.21.15.83:6816","nonce":30852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":30852},{"type":"v1","addr":"172.21.15.83:6823","nonce":30852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":30852},{"type":"v1","addr":"172.21.15.83:6820","nonce":30852}]},"public_addr":"172.21.15.83:6812/30852","cluster_addr":"172.21.15.83:6816/30852","heartbeat_back_addr":"172.21.15.83:6823/30852","heartbeat_front_addr":"172.21.15.83:6820/30852","state":["exists","up"]},{"osd":3,"uuid":"a24e3b64-a290-479e-ab17-c35a891960c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":30961},{"type":"v1","addr":"172.21.15.136:6801","nonce":30961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":30961},{"type":"v1","addr":"172.21.15.136:6803","nonce":30961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":30961},{"type":"v1","addr":"172.21.15.136:6811","nonce":30961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":30961},{"type":"v1","addr":"172.21.15.136:6807","nonce":30961}]},"public_addr":"172.21.15.136:6801/30961","cluster_addr":"172.21.15.136:6803/30961","heartbeat_back_addr":"172.21.15.136:6811/30961","heartbeat_front_addr":"172.21.15.136:6807/30961","state":["exists","up"]},{"osd":4,"uuid":"f3f49f6a-7a3d-4985-a8c9-995777f7fda5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":30963},{"type":"v1","addr":"172.21.15.136:6817","nonce":30963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":30963},{"type":"v1","addr":"172.21.15.136:6819","nonce":30963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":30963},{"type":"v1","addr":"172.21.15.136:6823","nonce":30963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":30963},{"type":"v1","addr":"172.21.15.136:6821","nonce":30963}]},"public_addr":"172.21.15.136:6817/30963","cluster_addr":"172.21.15.136:6819/30963","heartbeat_back_addr":"172.21.15.136:6823/30963","heartbeat_front_addr":"172.21.15.136:6821/30963","state":["exists","up"]},{"osd":5,"uuid":"ee8f2910-b899-4798-a9d8-762ea0e7c9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":30962},{"type":"v1","addr":"172.21.15.136:6806","nonce":30962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":30962},{"type":"v1","addr":"172.21.15.136:6810","nonce":30962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":30962},{"type":"v1","addr":"172.21.15.136:6815","nonce":30962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":30962},{"type":"v1","addr":"172.21.15.136:6813","nonce":30962}]},"public_addr":"172.21.15.136:6806/30962","cluster_addr":"172.21.15.136:6810/30962","heartbeat_back_addr":"172.21.15.136:6815/30962","heartbeat_front_addr":"172.21.15.136:6813/30962","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.810281+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.823140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.962397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.892268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:08.069943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:08.018060+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T01:48:10.519 DEBUG:teuthology.misc:6 of 6 OSDs are up 2023-11-16T01:48:10.519 INFO:tasks.ceph:Creating RBD pool 2023-11-16T01:48:10.519 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd pool create rbd 8 2023-11-16T01:48:11.315 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'rbd' created 2023-11-16T01:48:11.327 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-16T01:48:12.316 INFO:teuthology.orchestra.run.smithi083.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-16T01:48:12.329 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-16T01:48:12.330 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-16T01:48:12.330 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-16T01:48:12.330 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-16T01:48:12.330 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-16T01:48:12.651 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:48:12.662 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":30706},{"type":"v1","addr":"172.21.15.136:6825","nonce":30706}]},"active_addr":"172.21.15.136:6825/30706","active_change":"2023-11-16T01:48:05.222387+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":3120750667}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":999460619}]}]} 2023-11-16T01:48:12.664 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-16T01:48:12.664 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-16T01:48:12.664 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T01:48:12.946 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:48:12.946 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","created":"2023-11-16T01:47:58.606709+0000","modified":"2023-11-16T01:48:12.298782+0000","last_up_change":"2023-11-16T01:48:09.279683+0000","last_in_change":"2023-11-16T01:48:02.279946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-16T01:48:09.347564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-16T01:48:10.852819+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9244fa2e-56f6-41bf-b5e8-264183343a6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":30853},{"type":"v1","addr":"172.21.15.83:6801","nonce":30853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":30853},{"type":"v1","addr":"172.21.15.83:6803","nonce":30853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":30853},{"type":"v1","addr":"172.21.15.83:6807","nonce":30853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":30853},{"type":"v1","addr":"172.21.15.83:6805","nonce":30853}]},"public_addr":"172.21.15.83:6801/30853","cluster_addr":"172.21.15.83:6803/30853","heartbeat_back_addr":"172.21.15.83:6807/30853","heartbeat_front_addr":"172.21.15.83:6805/30853","state":["exists","up"]},{"osd":1,"uuid":"ed5dde4d-7737-428e-af42-a56e6224529d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":30851},{"type":"v1","addr":"172.21.15.83:6809","nonce":30851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6811","nonce":30851},{"type":"v1","addr":"172.21.15.83:6813","nonce":30851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6819","nonce":30851},{"type":"v1","addr":"172.21.15.83:6821","nonce":30851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":30851},{"type":"v1","addr":"172.21.15.83:6817","nonce":30851}]},"public_addr":"172.21.15.83:6809/30851","cluster_addr":"172.21.15.83:6813/30851","heartbeat_back_addr":"172.21.15.83:6821/30851","heartbeat_front_addr":"172.21.15.83:6817/30851","state":["exists","up"]},{"osd":2,"uuid":"b363e0df-3ed6-4fd7-b06a-6fc5f880f606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":30852},{"type":"v1","addr":"172.21.15.83:6812","nonce":30852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":30852},{"type":"v1","addr":"172.21.15.83:6816","nonce":30852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":30852},{"type":"v1","addr":"172.21.15.83:6823","nonce":30852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":30852},{"type":"v1","addr":"172.21.15.83:6820","nonce":30852}]},"public_addr":"172.21.15.83:6812/30852","cluster_addr":"172.21.15.83:6816/30852","heartbeat_back_addr":"172.21.15.83:6823/30852","heartbeat_front_addr":"172.21.15.83:6820/30852","state":["exists","up"]},{"osd":3,"uuid":"a24e3b64-a290-479e-ab17-c35a891960c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":30961},{"type":"v1","addr":"172.21.15.136:6801","nonce":30961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":30961},{"type":"v1","addr":"172.21.15.136:6803","nonce":30961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":30961},{"type":"v1","addr":"172.21.15.136:6811","nonce":30961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":30961},{"type":"v1","addr":"172.21.15.136:6807","nonce":30961}]},"public_addr":"172.21.15.136:6801/30961","cluster_addr":"172.21.15.136:6803/30961","heartbeat_back_addr":"172.21.15.136:6811/30961","heartbeat_front_addr":"172.21.15.136:6807/30961","state":["exists","up"]},{"osd":4,"uuid":"f3f49f6a-7a3d-4985-a8c9-995777f7fda5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":30963},{"type":"v1","addr":"172.21.15.136:6817","nonce":30963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":30963},{"type":"v1","addr":"172.21.15.136:6819","nonce":30963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":30963},{"type":"v1","addr":"172.21.15.136:6823","nonce":30963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":30963},{"type":"v1","addr":"172.21.15.136:6821","nonce":30963}]},"public_addr":"172.21.15.136:6817/30963","cluster_addr":"172.21.15.136:6819/30963","heartbeat_back_addr":"172.21.15.136:6823/30963","heartbeat_front_addr":"172.21.15.136:6821/30963","state":["exists","up"]},{"osd":5,"uuid":"ee8f2910-b899-4798-a9d8-762ea0e7c9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":30962},{"type":"v1","addr":"172.21.15.136:6806","nonce":30962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":30962},{"type":"v1","addr":"172.21.15.136:6810","nonce":30962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":30962},{"type":"v1","addr":"172.21.15.136:6815","nonce":30962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":30962},{"type":"v1","addr":"172.21.15.136:6813","nonce":30962}]},"public_addr":"172.21.15.136:6806/30962","cluster_addr":"172.21.15.136:6810/30962","heartbeat_back_addr":"172.21.15.136:6815/30962","heartbeat_front_addr":"172.21.15.136:6813/30962","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.810281+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.823140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.962397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.892268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:08.069943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:08.018060+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T01:48:12.959 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-16T01:48:12.959 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T01:48:13.241 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:48:13.241 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","created":"2023-11-16T01:47:58.606709+0000","modified":"2023-11-16T01:48:12.298782+0000","last_up_change":"2023-11-16T01:48:09.279683+0000","last_in_change":"2023-11-16T01:48:02.279946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-16T01:48:09.347564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-16T01:48:10.852819+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9244fa2e-56f6-41bf-b5e8-264183343a6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":30853},{"type":"v1","addr":"172.21.15.83:6801","nonce":30853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":30853},{"type":"v1","addr":"172.21.15.83:6803","nonce":30853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":30853},{"type":"v1","addr":"172.21.15.83:6807","nonce":30853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":30853},{"type":"v1","addr":"172.21.15.83:6805","nonce":30853}]},"public_addr":"172.21.15.83:6801/30853","cluster_addr":"172.21.15.83:6803/30853","heartbeat_back_addr":"172.21.15.83:6807/30853","heartbeat_front_addr":"172.21.15.83:6805/30853","state":["exists","up"]},{"osd":1,"uuid":"ed5dde4d-7737-428e-af42-a56e6224529d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":30851},{"type":"v1","addr":"172.21.15.83:6809","nonce":30851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6811","nonce":30851},{"type":"v1","addr":"172.21.15.83:6813","nonce":30851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6819","nonce":30851},{"type":"v1","addr":"172.21.15.83:6821","nonce":30851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":30851},{"type":"v1","addr":"172.21.15.83:6817","nonce":30851}]},"public_addr":"172.21.15.83:6809/30851","cluster_addr":"172.21.15.83:6813/30851","heartbeat_back_addr":"172.21.15.83:6821/30851","heartbeat_front_addr":"172.21.15.83:6817/30851","state":["exists","up"]},{"osd":2,"uuid":"b363e0df-3ed6-4fd7-b06a-6fc5f880f606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":30852},{"type":"v1","addr":"172.21.15.83:6812","nonce":30852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":30852},{"type":"v1","addr":"172.21.15.83:6816","nonce":30852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":30852},{"type":"v1","addr":"172.21.15.83:6823","nonce":30852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":30852},{"type":"v1","addr":"172.21.15.83:6820","nonce":30852}]},"public_addr":"172.21.15.83:6812/30852","cluster_addr":"172.21.15.83:6816/30852","heartbeat_back_addr":"172.21.15.83:6823/30852","heartbeat_front_addr":"172.21.15.83:6820/30852","state":["exists","up"]},{"osd":3,"uuid":"a24e3b64-a290-479e-ab17-c35a891960c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":30961},{"type":"v1","addr":"172.21.15.136:6801","nonce":30961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":30961},{"type":"v1","addr":"172.21.15.136:6803","nonce":30961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":30961},{"type":"v1","addr":"172.21.15.136:6811","nonce":30961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":30961},{"type":"v1","addr":"172.21.15.136:6807","nonce":30961}]},"public_addr":"172.21.15.136:6801/30961","cluster_addr":"172.21.15.136:6803/30961","heartbeat_back_addr":"172.21.15.136:6811/30961","heartbeat_front_addr":"172.21.15.136:6807/30961","state":["exists","up"]},{"osd":4,"uuid":"f3f49f6a-7a3d-4985-a8c9-995777f7fda5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":30963},{"type":"v1","addr":"172.21.15.136:6817","nonce":30963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":30963},{"type":"v1","addr":"172.21.15.136:6819","nonce":30963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":30963},{"type":"v1","addr":"172.21.15.136:6823","nonce":30963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":30963},{"type":"v1","addr":"172.21.15.136:6821","nonce":30963}]},"public_addr":"172.21.15.136:6817/30963","cluster_addr":"172.21.15.136:6819/30963","heartbeat_back_addr":"172.21.15.136:6823/30963","heartbeat_front_addr":"172.21.15.136:6821/30963","state":["exists","up"]},{"osd":5,"uuid":"ee8f2910-b899-4798-a9d8-762ea0e7c9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":30962},{"type":"v1","addr":"172.21.15.136:6806","nonce":30962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":30962},{"type":"v1","addr":"172.21.15.136:6810","nonce":30962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":30962},{"type":"v1","addr":"172.21.15.136:6815","nonce":30962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":30962},{"type":"v1","addr":"172.21.15.136:6813","nonce":30962}]},"public_addr":"172.21.15.136:6806/30962","cluster_addr":"172.21.15.136:6810/30962","heartbeat_back_addr":"172.21.15.136:6815/30962","heartbeat_front_addr":"172.21.15.136:6813/30962","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.810281+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.823140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.962397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.892268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:08.069943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:08.018060+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T01:48:13.255 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-16T01:48:13.419 INFO:teuthology.orchestra.run.smithi083.stdout:47244640258 2023-11-16T01:48:13.421 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-16T01:48:13.589 INFO:teuthology.orchestra.run.smithi083.stdout:47244640259 2023-11-16T01:48:13.590 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-16T01:48:13.956 INFO:teuthology.orchestra.run.smithi083.stdout:47244640259 2023-11-16T01:48:13.957 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-11-16T01:48:14.123 INFO:teuthology.orchestra.run.smithi083.stdout:47244640258 2023-11-16T01:48:14.125 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-11-16T01:48:14.295 INFO:teuthology.orchestra.run.smithi083.stdout:47244640258 2023-11-16T01:48:14.295 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-11-16T01:48:14.458 INFO:teuthology.orchestra.run.smithi083.stdout:47244640259 2023-11-16T01:48:14.458 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-16T01:48:14.743 INFO:teuthology.orchestra.run.smithi083.stdout:47244640257 2023-11-16T01:48:14.757 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.0 2023-11-16T01:48:15.758 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-16T01:48:16.046 INFO:teuthology.orchestra.run.smithi083.stdout:47244640259 2023-11-16T01:48:16.064 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640259 for osd.0 2023-11-16T01:48:16.065 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-16T01:48:16.338 INFO:teuthology.orchestra.run.smithi083.stdout:47244640259 2023-11-16T01:48:16.352 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2023-11-16T01:48:16.352 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-16T01:48:16.627 INFO:teuthology.orchestra.run.smithi083.stdout:47244640259 2023-11-16T01:48:16.641 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.2 2023-11-16T01:48:16.642 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-16T01:48:16.919 INFO:teuthology.orchestra.run.smithi083.stdout:47244640258 2023-11-16T01:48:16.934 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.3 2023-11-16T01:48:16.934 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-16T01:48:17.213 INFO:teuthology.orchestra.run.smithi083.stdout:47244640258 2023-11-16T01:48:17.226 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.4 2023-11-16T01:48:17.227 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-16T01:48:17.510 INFO:teuthology.orchestra.run.smithi083.stdout:47244640259 2023-11-16T01:48:17.523 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.5 2023-11-16T01:48:17.524 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-16T01:48:17.524 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T01:48:17.801 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:48:17.804 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2023-11-16T01:48:17.814 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-11-16T01:48:17.237955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1783384,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564447656,"statfs":{"total":579820584960,"available":577994399744,"internally_reserved":0,"allocated":827392,"data_stored":251135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.947226+0000","last_change":"2023-11-16T01:48:12.945654+0000","last_active":"2023-11-16T01:48:12.947226+0000","last_peered":"2023-11-16T01:48:12.947226+0000","last_clean":"2023-11-16T01:48:12.947226+0000","last_became_active":"2023-11-16T01:48:12.944985+0000","last_became_peered":"2023-11-16T01:48:12.944985+0000","last_unstale":"2023-11-16T01:48:12.947226+0000","last_undegraded":"2023-11-16T01:48:12.947226+0000","last_fullsized":"2023-11-16T01:48:12.947226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.317135+0000","last_change":"2023-11-16T01:48:12.315863+0000","last_active":"2023-11-16T01:48:12.317135+0000","last_peered":"2023-11-16T01:48:12.317135+0000","last_clean":"2023-11-16T01:48:12.317135+0000","last_became_active":"2023-11-16T01:48:12.315453+0000","last_became_peered":"2023-11-16T01:48:12.315453+0000","last_unstale":"2023-11-16T01:48:12.317135+0000","last_undegraded":"2023-11-16T01:48:12.317135+0000","last_fullsized":"2023-11-16T01:48:12.317135+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.742016+0000","last_change":"2023-11-16T01:48:12.740239+0000","last_active":"2023-11-16T01:48:12.742016+0000","last_peered":"2023-11-16T01:48:12.742016+0000","last_clean":"2023-11-16T01:48:12.742016+0000","last_became_active":"2023-11-16T01:48:12.739424+0000","last_became_peered":"2023-11-16T01:48:12.739424+0000","last_unstale":"2023-11-16T01:48:12.742016+0000","last_undegraded":"2023-11-16T01:48:12.742016+0000","last_fullsized":"2023-11-16T01:48:12.742016+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.741149+0000","last_change":"2023-11-16T01:48:12.740510+0000","last_active":"2023-11-16T01:48:12.741149+0000","last_peered":"2023-11-16T01:48:12.741149+0000","last_clean":"2023-11-16T01:48:12.741149+0000","last_became_active":"2023-11-16T01:48:12.739242+0000","last_became_peered":"2023-11-16T01:48:12.739242+0000","last_unstale":"2023-11-16T01:48:12.741149+0000","last_undegraded":"2023-11-16T01:48:12.741149+0000","last_fullsized":"2023-11-16T01:48:12.741149+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.317008+0000","last_change":"2023-11-16T01:48:12.315525+0000","last_active":"2023-11-16T01:48:12.317008+0000","last_peered":"2023-11-16T01:48:12.317008+0000","last_clean":"2023-11-16T01:48:12.317008+0000","last_became_active":"2023-11-16T01:48:12.314786+0000","last_became_peered":"2023-11-16T01:48:12.314786+0000","last_unstale":"2023-11-16T01:48:12.317008+0000","last_undegraded":"2023-11-16T01:48:12.317008+0000","last_fullsized":"2023-11-16T01:48:12.317008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.316326+0000","last_change":"2023-11-16T01:48:12.314855+0000","last_active":"2023-11-16T01:48:12.316326+0000","last_peered":"2023-11-16T01:48:12.316326+0000","last_clean":"2023-11-16T01:48:12.316326+0000","last_became_active":"2023-11-16T01:48:12.314447+0000","last_became_peered":"2023-11-16T01:48:12.314447+0000","last_unstale":"2023-11-16T01:48:12.316326+0000","last_undegraded":"2023-11-16T01:48:12.316326+0000","last_fullsized":"2023-11-16T01:48:12.316326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.316965+0000","last_change":"2023-11-16T01:48:12.314951+0000","last_active":"2023-11-16T01:48:12.316965+0000","last_peered":"2023-11-16T01:48:12.316965+0000","last_clean":"2023-11-16T01:48:12.316965+0000","last_became_active":"2023-11-16T01:48:12.314325+0000","last_became_peered":"2023-11-16T01:48:12.314325+0000","last_unstale":"2023-11-16T01:48:12.316965+0000","last_undegraded":"2023-11-16T01:48:12.316965+0000","last_fullsized":"2023-11-16T01:48:12.316965+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.317709+0000","last_change":"2023-11-16T01:48:12.316173+0000","last_active":"2023-11-16T01:48:12.317709+0000","last_peered":"2023-11-16T01:48:12.317709+0000","last_clean":"2023-11-16T01:48:12.317709+0000","last_became_active":"2023-11-16T01:48:12.315578+0000","last_became_peered":"2023-11-16T01:48:12.315578+0000","last_unstale":"2023-11-16T01:48:12.317709+0000","last_undegraded":"2023-11-16T01:48:12.317709+0000","last_fullsized":"2023-11-16T01:48:12.317709+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.307566+0000","last_change":"2023-11-16T01:48:11.314302+0000","last_active":"2023-11-16T01:48:12.307566+0000","last_peered":"2023-11-16T01:48:12.307566+0000","last_clean":"2023-11-16T01:48:12.307566+0000","last_became_active":"2023-11-16T01:48:11.313884+0000","last_became_peered":"2023-11-16T01:48:11.313884+0000","last_unstale":"2023-11-16T01:48:12.307566+0000","last_undegraded":"2023-11-16T01:48:12.307566+0000","last_fullsized":"2023-11-16T01:48:12.307566+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_clean_scrub_stamp":"2023-11-16T01:48:10.292390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":42924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":42924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297212,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074628,"statfs":{"total":96636764160,"available":96332419072,"internally_reserved":0,"allocated":118784,"data_stored":33608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T01:48:17.815 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T01:48:18.091 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:48:18.093 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2023-11-16T01:48:18.104 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-11-16T01:48:17.237955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1783384,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564447656,"statfs":{"total":579820584960,"available":577994399744,"internally_reserved":0,"allocated":827392,"data_stored":251135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.947226+0000","last_change":"2023-11-16T01:48:12.945654+0000","last_active":"2023-11-16T01:48:12.947226+0000","last_peered":"2023-11-16T01:48:12.947226+0000","last_clean":"2023-11-16T01:48:12.947226+0000","last_became_active":"2023-11-16T01:48:12.944985+0000","last_became_peered":"2023-11-16T01:48:12.944985+0000","last_unstale":"2023-11-16T01:48:12.947226+0000","last_undegraded":"2023-11-16T01:48:12.947226+0000","last_fullsized":"2023-11-16T01:48:12.947226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.317135+0000","last_change":"2023-11-16T01:48:12.315863+0000","last_active":"2023-11-16T01:48:12.317135+0000","last_peered":"2023-11-16T01:48:12.317135+0000","last_clean":"2023-11-16T01:48:12.317135+0000","last_became_active":"2023-11-16T01:48:12.315453+0000","last_became_peered":"2023-11-16T01:48:12.315453+0000","last_unstale":"2023-11-16T01:48:12.317135+0000","last_undegraded":"2023-11-16T01:48:12.317135+0000","last_fullsized":"2023-11-16T01:48:12.317135+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.742016+0000","last_change":"2023-11-16T01:48:12.740239+0000","last_active":"2023-11-16T01:48:12.742016+0000","last_peered":"2023-11-16T01:48:12.742016+0000","last_clean":"2023-11-16T01:48:12.742016+0000","last_became_active":"2023-11-16T01:48:12.739424+0000","last_became_peered":"2023-11-16T01:48:12.739424+0000","last_unstale":"2023-11-16T01:48:12.742016+0000","last_undegraded":"2023-11-16T01:48:12.742016+0000","last_fullsized":"2023-11-16T01:48:12.742016+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.741149+0000","last_change":"2023-11-16T01:48:12.740510+0000","last_active":"2023-11-16T01:48:12.741149+0000","last_peered":"2023-11-16T01:48:12.741149+0000","last_clean":"2023-11-16T01:48:12.741149+0000","last_became_active":"2023-11-16T01:48:12.739242+0000","last_became_peered":"2023-11-16T01:48:12.739242+0000","last_unstale":"2023-11-16T01:48:12.741149+0000","last_undegraded":"2023-11-16T01:48:12.741149+0000","last_fullsized":"2023-11-16T01:48:12.741149+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.317008+0000","last_change":"2023-11-16T01:48:12.315525+0000","last_active":"2023-11-16T01:48:12.317008+0000","last_peered":"2023-11-16T01:48:12.317008+0000","last_clean":"2023-11-16T01:48:12.317008+0000","last_became_active":"2023-11-16T01:48:12.314786+0000","last_became_peered":"2023-11-16T01:48:12.314786+0000","last_unstale":"2023-11-16T01:48:12.317008+0000","last_undegraded":"2023-11-16T01:48:12.317008+0000","last_fullsized":"2023-11-16T01:48:12.317008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.316326+0000","last_change":"2023-11-16T01:48:12.314855+0000","last_active":"2023-11-16T01:48:12.316326+0000","last_peered":"2023-11-16T01:48:12.316326+0000","last_clean":"2023-11-16T01:48:12.316326+0000","last_became_active":"2023-11-16T01:48:12.314447+0000","last_became_peered":"2023-11-16T01:48:12.314447+0000","last_unstale":"2023-11-16T01:48:12.316326+0000","last_undegraded":"2023-11-16T01:48:12.316326+0000","last_fullsized":"2023-11-16T01:48:12.316326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.316965+0000","last_change":"2023-11-16T01:48:12.314951+0000","last_active":"2023-11-16T01:48:12.316965+0000","last_peered":"2023-11-16T01:48:12.316965+0000","last_clean":"2023-11-16T01:48:12.316965+0000","last_became_active":"2023-11-16T01:48:12.314325+0000","last_became_peered":"2023-11-16T01:48:12.314325+0000","last_unstale":"2023-11-16T01:48:12.316965+0000","last_undegraded":"2023-11-16T01:48:12.316965+0000","last_fullsized":"2023-11-16T01:48:12.316965+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.317709+0000","last_change":"2023-11-16T01:48:12.316173+0000","last_active":"2023-11-16T01:48:12.317709+0000","last_peered":"2023-11-16T01:48:12.317709+0000","last_clean":"2023-11-16T01:48:12.317709+0000","last_became_active":"2023-11-16T01:48:12.315578+0000","last_became_peered":"2023-11-16T01:48:12.315578+0000","last_unstale":"2023-11-16T01:48:12.317709+0000","last_undegraded":"2023-11-16T01:48:12.317709+0000","last_fullsized":"2023-11-16T01:48:12.317709+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.307566+0000","last_change":"2023-11-16T01:48:11.314302+0000","last_active":"2023-11-16T01:48:12.307566+0000","last_peered":"2023-11-16T01:48:12.307566+0000","last_clean":"2023-11-16T01:48:12.307566+0000","last_became_active":"2023-11-16T01:48:11.313884+0000","last_became_peered":"2023-11-16T01:48:11.313884+0000","last_unstale":"2023-11-16T01:48:12.307566+0000","last_undegraded":"2023-11-16T01:48:12.307566+0000","last_fullsized":"2023-11-16T01:48:12.307566+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_clean_scrub_stamp":"2023-11-16T01:48:10.292390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":42924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":42924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297212,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074628,"statfs":{"total":96636764160,"available":96332419072,"internally_reserved":0,"allocated":118784,"data_stored":33608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T01:48:18.105 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-16T01:48:18.106 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-16T01:48:18.106 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-16T01:48:18.106 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-16T01:48:18.418 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:48:18.432 INFO:teuthology.orchestra.run.smithi083.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-16T01:48:18.432 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-16T01:48:18.432 INFO:teuthology.run_tasks:Running task mon_thrash... 2023-11-16T01:48:18.455 INFO:tasks.mon_thrash:Beginning mon_thrash... 2023-11-16T01:48:18.455 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T01:48:18.746 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:48:18.747 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":15,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","created":"2023-11-16T01:47:58.606709+0000","modified":"2023-11-16T01:48:13.301809+0000","last_up_change":"2023-11-16T01:48:09.279683+0000","last_in_change":"2023-11-16T01:48:02.279946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-16T01:48:09.347564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-16T01:48:10.852819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9244fa2e-56f6-41bf-b5e8-264183343a6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":30853},{"type":"v1","addr":"172.21.15.83:6801","nonce":30853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":30853},{"type":"v1","addr":"172.21.15.83:6803","nonce":30853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":30853},{"type":"v1","addr":"172.21.15.83:6807","nonce":30853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":30853},{"type":"v1","addr":"172.21.15.83:6805","nonce":30853}]},"public_addr":"172.21.15.83:6801/30853","cluster_addr":"172.21.15.83:6803/30853","heartbeat_back_addr":"172.21.15.83:6807/30853","heartbeat_front_addr":"172.21.15.83:6805/30853","state":["exists","up"]},{"osd":1,"uuid":"ed5dde4d-7737-428e-af42-a56e6224529d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":30851},{"type":"v1","addr":"172.21.15.83:6809","nonce":30851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6811","nonce":30851},{"type":"v1","addr":"172.21.15.83:6813","nonce":30851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6819","nonce":30851},{"type":"v1","addr":"172.21.15.83:6821","nonce":30851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":30851},{"type":"v1","addr":"172.21.15.83:6817","nonce":30851}]},"public_addr":"172.21.15.83:6809/30851","cluster_addr":"172.21.15.83:6813/30851","heartbeat_back_addr":"172.21.15.83:6821/30851","heartbeat_front_addr":"172.21.15.83:6817/30851","state":["exists","up"]},{"osd":2,"uuid":"b363e0df-3ed6-4fd7-b06a-6fc5f880f606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":30852},{"type":"v1","addr":"172.21.15.83:6812","nonce":30852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":30852},{"type":"v1","addr":"172.21.15.83:6816","nonce":30852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":30852},{"type":"v1","addr":"172.21.15.83:6823","nonce":30852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":30852},{"type":"v1","addr":"172.21.15.83:6820","nonce":30852}]},"public_addr":"172.21.15.83:6812/30852","cluster_addr":"172.21.15.83:6816/30852","heartbeat_back_addr":"172.21.15.83:6823/30852","heartbeat_front_addr":"172.21.15.83:6820/30852","state":["exists","up"]},{"osd":3,"uuid":"a24e3b64-a290-479e-ab17-c35a891960c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":30961},{"type":"v1","addr":"172.21.15.136:6801","nonce":30961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":30961},{"type":"v1","addr":"172.21.15.136:6803","nonce":30961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":30961},{"type":"v1","addr":"172.21.15.136:6811","nonce":30961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":30961},{"type":"v1","addr":"172.21.15.136:6807","nonce":30961}]},"public_addr":"172.21.15.136:6801/30961","cluster_addr":"172.21.15.136:6803/30961","heartbeat_back_addr":"172.21.15.136:6811/30961","heartbeat_front_addr":"172.21.15.136:6807/30961","state":["exists","up"]},{"osd":4,"uuid":"f3f49f6a-7a3d-4985-a8c9-995777f7fda5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":30963},{"type":"v1","addr":"172.21.15.136:6817","nonce":30963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":30963},{"type":"v1","addr":"172.21.15.136:6819","nonce":30963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":30963},{"type":"v1","addr":"172.21.15.136:6823","nonce":30963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":30963},{"type":"v1","addr":"172.21.15.136:6821","nonce":30963}]},"public_addr":"172.21.15.136:6817/30963","cluster_addr":"172.21.15.136:6819/30963","heartbeat_back_addr":"172.21.15.136:6823/30963","heartbeat_front_addr":"172.21.15.136:6821/30963","state":["exists","up"]},{"osd":5,"uuid":"ee8f2910-b899-4798-a9d8-762ea0e7c9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":30962},{"type":"v1","addr":"172.21.15.136:6806","nonce":30962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":30962},{"type":"v1","addr":"172.21.15.136:6810","nonce":30962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":30962},{"type":"v1","addr":"172.21.15.136:6815","nonce":30962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":30962},{"type":"v1","addr":"172.21.15.136:6813","nonce":30962}]},"public_addr":"172.21.15.136:6806/30962","cluster_addr":"172.21.15.136:6810/30962","heartbeat_back_addr":"172.21.15.136:6815/30962","heartbeat_front_addr":"172.21.15.136:6813/30962","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.810281+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.823140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.962397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.892268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:08.069943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:08.018060+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T01:48:18.760 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-11-16T01:48:09.347564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-16T01:48:10.852819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-11-16T01:48:18.760 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2023-11-16T01:48:19.088 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2023-11-16T01:48:19.107 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-16T01:48:19.392 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 8 2023-11-16T01:48:19.405 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2023-11-16T01:48:19.406 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T01:48:19.681 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:48:19.683 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2023-11-16T01:48:19.694 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2023-11-16T01:48:19.238583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1783408,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564447632,"statfs":{"total":579820584960,"available":577994375168,"internally_reserved":0,"allocated":851968,"data_stored":261420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.947226+0000","last_change":"2023-11-16T01:48:12.945654+0000","last_active":"2023-11-16T01:48:12.947226+0000","last_peered":"2023-11-16T01:48:12.947226+0000","last_clean":"2023-11-16T01:48:12.947226+0000","last_became_active":"2023-11-16T01:48:12.944985+0000","last_became_peered":"2023-11-16T01:48:12.944985+0000","last_unstale":"2023-11-16T01:48:12.947226+0000","last_undegraded":"2023-11-16T01:48:12.947226+0000","last_fullsized":"2023-11-16T01:48:12.947226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.317135+0000","last_change":"2023-11-16T01:48:12.315863+0000","last_active":"2023-11-16T01:48:12.317135+0000","last_peered":"2023-11-16T01:48:12.317135+0000","last_clean":"2023-11-16T01:48:12.317135+0000","last_became_active":"2023-11-16T01:48:12.315453+0000","last_became_peered":"2023-11-16T01:48:12.315453+0000","last_unstale":"2023-11-16T01:48:12.317135+0000","last_undegraded":"2023-11-16T01:48:12.317135+0000","last_fullsized":"2023-11-16T01:48:12.317135+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.742016+0000","last_change":"2023-11-16T01:48:12.740239+0000","last_active":"2023-11-16T01:48:12.742016+0000","last_peered":"2023-11-16T01:48:12.742016+0000","last_clean":"2023-11-16T01:48:12.742016+0000","last_became_active":"2023-11-16T01:48:12.739424+0000","last_became_peered":"2023-11-16T01:48:12.739424+0000","last_unstale":"2023-11-16T01:48:12.742016+0000","last_undegraded":"2023-11-16T01:48:12.742016+0000","last_fullsized":"2023-11-16T01:48:12.742016+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.741149+0000","last_change":"2023-11-16T01:48:12.740510+0000","last_active":"2023-11-16T01:48:12.741149+0000","last_peered":"2023-11-16T01:48:12.741149+0000","last_clean":"2023-11-16T01:48:12.741149+0000","last_became_active":"2023-11-16T01:48:12.739242+0000","last_became_peered":"2023-11-16T01:48:12.739242+0000","last_unstale":"2023-11-16T01:48:12.741149+0000","last_undegraded":"2023-11-16T01:48:12.741149+0000","last_fullsized":"2023-11-16T01:48:12.741149+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.317008+0000","last_change":"2023-11-16T01:48:12.315525+0000","last_active":"2023-11-16T01:48:12.317008+0000","last_peered":"2023-11-16T01:48:12.317008+0000","last_clean":"2023-11-16T01:48:12.317008+0000","last_became_active":"2023-11-16T01:48:12.314786+0000","last_became_peered":"2023-11-16T01:48:12.314786+0000","last_unstale":"2023-11-16T01:48:12.317008+0000","last_undegraded":"2023-11-16T01:48:12.317008+0000","last_fullsized":"2023-11-16T01:48:12.317008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.316326+0000","last_change":"2023-11-16T01:48:12.314855+0000","last_active":"2023-11-16T01:48:12.316326+0000","last_peered":"2023-11-16T01:48:12.316326+0000","last_clean":"2023-11-16T01:48:12.316326+0000","last_became_active":"2023-11-16T01:48:12.314447+0000","last_became_peered":"2023-11-16T01:48:12.314447+0000","last_unstale":"2023-11-16T01:48:12.316326+0000","last_undegraded":"2023-11-16T01:48:12.316326+0000","last_fullsized":"2023-11-16T01:48:12.316326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.316965+0000","last_change":"2023-11-16T01:48:12.314951+0000","last_active":"2023-11-16T01:48:12.316965+0000","last_peered":"2023-11-16T01:48:12.316965+0000","last_clean":"2023-11-16T01:48:12.316965+0000","last_became_active":"2023-11-16T01:48:12.314325+0000","last_became_peered":"2023-11-16T01:48:12.314325+0000","last_unstale":"2023-11-16T01:48:12.316965+0000","last_undegraded":"2023-11-16T01:48:12.316965+0000","last_fullsized":"2023-11-16T01:48:12.316965+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.317709+0000","last_change":"2023-11-16T01:48:12.316173+0000","last_active":"2023-11-16T01:48:12.317709+0000","last_peered":"2023-11-16T01:48:12.317709+0000","last_clean":"2023-11-16T01:48:12.317709+0000","last_became_active":"2023-11-16T01:48:12.315578+0000","last_became_peered":"2023-11-16T01:48:12.315578+0000","last_unstale":"2023-11-16T01:48:12.317709+0000","last_undegraded":"2023-11-16T01:48:12.317709+0000","last_fullsized":"2023-11-16T01:48:12.317709+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.307566+0000","last_change":"2023-11-16T01:48:11.314302+0000","last_active":"2023-11-16T01:48:12.307566+0000","last_peered":"2023-11-16T01:48:12.307566+0000","last_clean":"2023-11-16T01:48:12.307566+0000","last_became_active":"2023-11-16T01:48:11.313884+0000","last_became_peered":"2023-11-16T01:48:11.313884+0000","last_unstale":"2023-11-16T01:48:12.307566+0000","last_undegraded":"2023-11-16T01:48:12.307566+0000","last_fullsized":"2023-11-16T01:48:12.307566+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_clean_scrub_stamp":"2023-11-16T01:48:10.292390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":42924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":42924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T01:48:19.695 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T01:48:19.970 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:48:19.973 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2023-11-16T01:48:19.983 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2023-11-16T01:48:19.238583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1783408,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564447632,"statfs":{"total":579820584960,"available":577994375168,"internally_reserved":0,"allocated":851968,"data_stored":261420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.947226+0000","last_change":"2023-11-16T01:48:12.945654+0000","last_active":"2023-11-16T01:48:12.947226+0000","last_peered":"2023-11-16T01:48:12.947226+0000","last_clean":"2023-11-16T01:48:12.947226+0000","last_became_active":"2023-11-16T01:48:12.944985+0000","last_became_peered":"2023-11-16T01:48:12.944985+0000","last_unstale":"2023-11-16T01:48:12.947226+0000","last_undegraded":"2023-11-16T01:48:12.947226+0000","last_fullsized":"2023-11-16T01:48:12.947226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.317135+0000","last_change":"2023-11-16T01:48:12.315863+0000","last_active":"2023-11-16T01:48:12.317135+0000","last_peered":"2023-11-16T01:48:12.317135+0000","last_clean":"2023-11-16T01:48:12.317135+0000","last_became_active":"2023-11-16T01:48:12.315453+0000","last_became_peered":"2023-11-16T01:48:12.315453+0000","last_unstale":"2023-11-16T01:48:12.317135+0000","last_undegraded":"2023-11-16T01:48:12.317135+0000","last_fullsized":"2023-11-16T01:48:12.317135+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.742016+0000","last_change":"2023-11-16T01:48:12.740239+0000","last_active":"2023-11-16T01:48:12.742016+0000","last_peered":"2023-11-16T01:48:12.742016+0000","last_clean":"2023-11-16T01:48:12.742016+0000","last_became_active":"2023-11-16T01:48:12.739424+0000","last_became_peered":"2023-11-16T01:48:12.739424+0000","last_unstale":"2023-11-16T01:48:12.742016+0000","last_undegraded":"2023-11-16T01:48:12.742016+0000","last_fullsized":"2023-11-16T01:48:12.742016+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.741149+0000","last_change":"2023-11-16T01:48:12.740510+0000","last_active":"2023-11-16T01:48:12.741149+0000","last_peered":"2023-11-16T01:48:12.741149+0000","last_clean":"2023-11-16T01:48:12.741149+0000","last_became_active":"2023-11-16T01:48:12.739242+0000","last_became_peered":"2023-11-16T01:48:12.739242+0000","last_unstale":"2023-11-16T01:48:12.741149+0000","last_undegraded":"2023-11-16T01:48:12.741149+0000","last_fullsized":"2023-11-16T01:48:12.741149+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.317008+0000","last_change":"2023-11-16T01:48:12.315525+0000","last_active":"2023-11-16T01:48:12.317008+0000","last_peered":"2023-11-16T01:48:12.317008+0000","last_clean":"2023-11-16T01:48:12.317008+0000","last_became_active":"2023-11-16T01:48:12.314786+0000","last_became_peered":"2023-11-16T01:48:12.314786+0000","last_unstale":"2023-11-16T01:48:12.317008+0000","last_undegraded":"2023-11-16T01:48:12.317008+0000","last_fullsized":"2023-11-16T01:48:12.317008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.316326+0000","last_change":"2023-11-16T01:48:12.314855+0000","last_active":"2023-11-16T01:48:12.316326+0000","last_peered":"2023-11-16T01:48:12.316326+0000","last_clean":"2023-11-16T01:48:12.316326+0000","last_became_active":"2023-11-16T01:48:12.314447+0000","last_became_peered":"2023-11-16T01:48:12.314447+0000","last_unstale":"2023-11-16T01:48:12.316326+0000","last_undegraded":"2023-11-16T01:48:12.316326+0000","last_fullsized":"2023-11-16T01:48:12.316326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.316965+0000","last_change":"2023-11-16T01:48:12.314951+0000","last_active":"2023-11-16T01:48:12.316965+0000","last_peered":"2023-11-16T01:48:12.316965+0000","last_clean":"2023-11-16T01:48:12.316965+0000","last_became_active":"2023-11-16T01:48:12.314325+0000","last_became_peered":"2023-11-16T01:48:12.314325+0000","last_unstale":"2023-11-16T01:48:12.316965+0000","last_undegraded":"2023-11-16T01:48:12.316965+0000","last_fullsized":"2023-11-16T01:48:12.316965+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.317709+0000","last_change":"2023-11-16T01:48:12.316173+0000","last_active":"2023-11-16T01:48:12.317709+0000","last_peered":"2023-11-16T01:48:12.317709+0000","last_clean":"2023-11-16T01:48:12.317709+0000","last_became_active":"2023-11-16T01:48:12.315578+0000","last_became_peered":"2023-11-16T01:48:12.315578+0000","last_unstale":"2023-11-16T01:48:12.317709+0000","last_undegraded":"2023-11-16T01:48:12.317709+0000","last_fullsized":"2023-11-16T01:48:12.317709+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-16T01:48:12.307566+0000","last_change":"2023-11-16T01:48:11.314302+0000","last_active":"2023-11-16T01:48:12.307566+0000","last_peered":"2023-11-16T01:48:12.307566+0000","last_clean":"2023-11-16T01:48:12.307566+0000","last_became_active":"2023-11-16T01:48:11.313884+0000","last_became_peered":"2023-11-16T01:48:11.313884+0000","last_unstale":"2023-11-16T01:48:12.307566+0000","last_undegraded":"2023-11-16T01:48:12.307566+0000","last_fullsized":"2023-11-16T01:48:12.307566+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_clean_scrub_stamp":"2023-11-16T01:48:10.292390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":42924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":42924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297236,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074604,"statfs":{"total":96636764160,"available":96332394496,"internally_reserved":0,"allocated":143360,"data_stored":43893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T01:48:19.985 INFO:tasks.mon_thrash.ceph_manager:clean! 2023-11-16T01:48:19.986 DEBUG:tasks.mon_thrash:Yielding 2023-11-16T01:48:19.986 INFO:teuthology.run_tasks:Running task workunit... 2023-11-16T01:48:20.001 INFO:tasks.workunit:Pulling workunits from ref ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-16T01:48:20.001 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-11-16T01:48:20.001 DEBUG:teuthology.orchestra.run.smithi136:> stat -- /home/ubuntu/cephtest/mnt.0 2023-11-16T01:48:20.002 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2023-11-16T01:48:20.002 INFO:tasks.mon_thrash.mon_thrasher:seed: 1700099299, revive delay: 20.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2023-11-16T01:48:20.002 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-16T01:48:20.002 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:48:20.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:48:20.011 INFO:teuthology.orchestra.run.smithi136.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-11-16T01:48:20.011 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-11-16T01:48:20.061 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-11-16T01:48:20.061 DEBUG:teuthology.orchestra.run.smithi136:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-11-16T01:48:20.069 INFO:tasks.workunit:timeout=3h 2023-11-16T01:48:20.069 INFO:tasks.workunit:cleanup=True 2023-11-16T01:48:20.069 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-16T01:48:20.078 INFO:tasks.workunit.client.0.smithi136.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-11-16T01:48:20.339 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:48:20.353 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-16T01:48:20.353 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-11-16T01:48:20.353 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-16T01:48:20.353 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-16T01:48:20.487 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:48:20.488 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:48:20.488 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:48:20.488 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "leader", 2023-11-16T01:48:20.488 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 8, 2023-11-16T01:48:20.488 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:48:20.488 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:48:20.488 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:48:20.488 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:48:20.488 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:20.488 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 21, 2023-11-16T01:48:20.488 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:48:20.489 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:48:20.489 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:48:20.489 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:48:20.489 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:48:20.489 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:48:20.489 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:48:20.489 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:48:20.489 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:48:20.489 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:48:20.489 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:48:20.489 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:20.489 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:48:20.490 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:48:20.490 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:48:20.490 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:48:20.490 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:48:20.490 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:48:20.490 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:48:20.490 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:48:20.490 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:48:20.490 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:48:20.490 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.490 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.490 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:48:20.491 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:48:20.491 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:48:20.491 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:48:20.491 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:48:20.491 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:48:20.491 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:48:20.491 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:48:20.491 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:48:20.491 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:48:20.491 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:48:20.491 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:48:20.491 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:48:20.492 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:48:20.492 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:48:20.492 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:48:20.492 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:48:20.492 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:48:20.492 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:48:20.492 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:48:20.492 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:48:20.492 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:48:20.492 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:48:20.492 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:48:20.492 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:48:20.493 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:20.493 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:48:20.493 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.493 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:48:20.493 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.493 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:48:20.493 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:48:20.493 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:48:20.493 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:48:20.493 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.493 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:48:20.493 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:48:20.494 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.494 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.494 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.494 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:48:20.494 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:48:20.494 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.494 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.494 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.494 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.494 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:48:20.494 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:48:20.494 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:48:20.495 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:48:20.495 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:48:20.495 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.495 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.495 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:48:20.495 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:48:20.495 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:48:20.495 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:48:20.495 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.495 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:48:20.495 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:48:20.495 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.496 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.496 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.496 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:48:20.496 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:48:20.496 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.496 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.496 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.496 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.496 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:48:20.496 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:48:20.496 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:48:20.496 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:48:20.497 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:48:20.497 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.497 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.497 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:48:20.497 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:48:20.497 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:48:20.497 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:48:20.497 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.497 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:48:20.497 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:48:20.497 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.497 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.497 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.498 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:48:20.498 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:48:20.498 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.498 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.498 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.498 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.498 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:48:20.498 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:48:20.498 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:48:20.498 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:48:20.498 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:48:20.499 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.499 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.499 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.499 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:48:20.499 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:48:20.499 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.499 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:48:20.499 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:48:20.499 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:48:20.499 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.499 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:20.501 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:48:20.501 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.501 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:48:20.501 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:48:20.501 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 3 2023-11-16T01:48:20.501 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.501 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:20.501 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:48:20.502 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.502 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:48:20.502 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:48:20.502 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:48:20.502 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.502 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.502 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.502 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:48:20.502 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:48:20.503 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:48:20.503 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-16T01:48:20.504 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-16T01:48:20.649 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:48:20.649 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:48:20.649 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:48:20.650 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:48:20.650 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 8, 2023-11-16T01:48:20.650 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:48:20.650 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:48:20.650 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:48:20.650 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:48:20.650 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:20.650 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 22, 2023-11-16T01:48:20.650 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:48:20.650 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:48:20.650 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:48:20.651 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:48:20.651 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:48:20.651 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:48:20.651 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:48:20.651 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:48:20.651 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:48:20.651 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:48:20.651 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:48:20.651 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:20.651 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:48:20.651 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:48:20.651 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:48:20.652 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:48:20.652 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:48:20.652 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:48:20.652 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:48:20.652 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:48:20.652 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:48:20.652 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:48:20.652 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.652 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.652 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:48:20.652 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:48:20.652 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:48:20.653 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:48:20.653 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:48:20.653 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:48:20.653 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:48:20.653 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:48:20.653 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:48:20.653 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:48:20.653 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:48:20.653 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:48:20.653 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:48:20.653 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:48:20.653 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:48:20.654 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:48:20.654 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:48:20.654 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:48:20.654 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:48:20.654 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:48:20.654 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:48:20.654 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:48:20.654 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:48:20.654 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:48:20.654 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:48:20.654 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:20.654 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:48:20.655 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.655 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:48:20.655 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.655 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:48:20.655 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:48:20.655 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:48:20.655 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:48:20.655 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.655 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:48:20.655 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:48:20.655 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.655 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.656 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.656 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:48:20.656 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:48:20.656 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.656 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.656 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.656 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.656 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:48:20.656 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:48:20.656 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:48:20.656 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:48:20.656 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:48:20.656 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.657 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.657 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:48:20.657 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:48:20.657 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:48:20.657 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:48:20.657 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.657 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:48:20.657 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:48:20.657 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.657 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.657 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.657 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:48:20.658 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:48:20.658 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.658 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.658 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.658 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.658 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:48:20.658 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:48:20.658 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:48:20.658 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:48:20.658 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:48:20.658 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.658 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.659 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:48:20.659 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:48:20.659 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:48:20.659 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:48:20.659 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.659 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:48:20.659 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:48:20.659 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.659 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.659 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.659 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:48:20.659 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:48:20.660 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.660 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.660 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.660 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.660 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:48:20.660 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:48:20.660 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:48:20.660 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:48:20.660 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:48:20.660 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.660 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.660 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.661 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:48:20.661 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:48:20.661 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.661 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:48:20.661 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:48:20.661 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:48:20.661 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.661 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:20.661 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:48:20.661 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.661 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:48:20.661 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:48:20.661 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:48:20.662 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.662 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.662 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.662 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:48:20.662 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:48:20.663 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:48:20.663 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-16T01:48:20.669 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-16T01:48:20.819 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:48:20.819 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:48:20.819 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:48:20.820 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:48:20.820 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 8, 2023-11-16T01:48:20.820 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:48:20.820 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:48:20.821 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:48:20.821 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:48:20.821 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:20.821 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 22, 2023-11-16T01:48:20.821 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:48:20.822 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:48:20.822 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:48:20.822 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:48:20.822 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:48:20.822 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:48:20.823 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:48:20.823 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:48:20.823 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:48:20.823 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:48:20.823 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:48:20.824 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:20.824 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:48:20.824 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:48:20.824 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:48:20.824 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:48:20.824 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:48:20.824 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:48:20.824 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:48:20.824 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:48:20.824 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:48:20.824 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:48:20.825 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.825 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.825 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:48:20.825 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:48:20.825 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:48:20.825 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:48:20.825 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:48:20.825 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:48:20.825 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:48:20.825 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:48:20.825 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:48:20.826 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:48:20.826 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:48:20.826 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:48:20.826 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:48:20.826 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:48:20.826 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:48:20.826 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:48:20.826 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:48:20.826 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:48:20.826 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:48:20.826 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:48:20.827 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:48:20.827 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:48:20.827 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:48:20.827 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:48:20.827 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:48:20.827 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:20.827 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:48:20.827 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.827 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:48:20.827 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.827 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:48:20.827 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:48:20.828 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:48:20.828 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:48:20.828 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.828 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:48:20.828 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:48:20.828 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.828 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.828 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.828 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:48:20.828 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:48:20.828 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.828 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.829 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.829 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.829 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:48:20.829 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:48:20.829 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:48:20.829 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:48:20.829 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:48:20.829 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.829 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.829 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:48:20.829 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:48:20.829 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:48:20.830 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:48:20.830 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.830 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:48:20.830 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:48:20.830 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.830 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.830 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.830 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:48:20.830 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:48:20.830 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.830 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.830 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.830 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.831 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:48:20.831 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:48:20.831 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:48:20.831 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:48:20.831 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:48:20.831 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.831 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.831 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:48:20.831 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:48:20.831 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:48:20.831 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:48:20.831 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.832 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:48:20.832 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:48:20.832 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.832 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.832 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.832 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:48:20.832 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:48:20.832 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:20.832 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.832 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.833 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.833 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:48:20.833 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:48:20.833 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:48:20.833 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:48:20.833 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:48:20.833 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.833 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.833 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.833 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:48:20.834 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:48:20.834 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.834 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:48:20.834 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:48:20.834 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:48:20.834 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.834 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:20.834 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:48:20.834 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.834 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:48:20.835 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:48:20.835 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 3 2023-11-16T01:48:20.835 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.835 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:20.835 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:48:20.835 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.835 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:48:20.835 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:48:20.835 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 3 2023-11-16T01:48:20.835 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.835 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:20.835 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": [ 2023-11-16T01:48:20.835 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:20.836 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:48:20.836 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:48:20.836 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:48:20.836 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:20.836 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:20.836 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:20.836 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:48:20.837 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:48:20.838 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:48:20.838 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-11-16T01:48:20.838 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-11-16T01:48:20.838 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-11-16T01:48:20.838 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-11-16T01:48:20.838 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-16T01:48:20.839 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T01:48:20.860 INFO:tasks.ceph.mon.b:Stopped 2023-11-16T01:48:20.860 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-11-16T01:48:20.860 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:48:21.194 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:48:24.208 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:48:25.167 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":26,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:48:28.182 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:48:28.524 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:48:31.539 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:48:35.366 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":12,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:48:35.380 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-11-16T01:48:35.380 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-11-16T01:48:35.380 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-16T01:48:35.517 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:48:35.518 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:48:35.518 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:48:35.518 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "leader", 2023-11-16T01:48:35.518 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 12, 2023-11-16T01:48:35.518 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:48:35.518 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:48:35.518 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:48:35.518 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:35.518 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 0, 2023-11-16T01:48:35.518 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:48:35.519 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:48:35.519 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:48:35.519 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:48:35.519 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:48:35.519 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:48:35.519 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:48:35.519 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:48:35.519 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:48:35.519 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:48:35.519 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:48:35.519 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:35.519 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:48:35.520 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:48:35.520 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:48:35.520 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:48:35.520 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:48:35.520 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:48:35.520 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:48:35.520 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:48:35.520 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:48:35.520 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:48:35.520 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:35.520 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.520 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:48:35.521 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:48:35.521 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:48:35.521 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:48:35.521 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:48:35.521 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:48:35.521 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:48:35.521 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:48:35.521 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:48:35.521 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:48:35.521 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:48:35.521 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:48:35.521 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:48:35.522 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:48:35.522 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:48:35.522 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:48:35.522 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:48:35.522 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:48:35.522 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:48:35.522 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:48:35.522 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:48:35.522 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:48:35.522 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:48:35.522 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:48:35.523 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:48:35.523 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:35.523 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:48:35.523 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.523 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:48:35.523 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.524 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:48:35.524 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:48:35.524 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:48:35.524 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:48:35.524 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.524 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:48:35.524 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:48:35.524 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:35.524 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.524 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.525 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:48:35.525 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:48:35.525 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:35.525 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:35.525 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:35.525 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.525 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:48:35.525 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:48:35.525 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:48:35.525 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:48:35.525 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:48:35.525 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.525 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.526 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:48:35.526 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:48:35.526 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:48:35.526 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:48:35.526 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.526 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:48:35.526 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:48:35.526 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:35.526 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.526 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.527 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:48:35.527 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:48:35.527 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:35.527 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:35.527 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:35.527 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.527 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:48:35.527 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:48:35.528 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:48:35.528 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:48:35.528 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:48:35.528 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.528 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.528 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:48:35.528 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:48:35.528 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:48:35.528 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:48:35.528 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.528 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:48:35.528 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:48:35.528 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:35.529 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.529 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.529 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:48:35.529 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:48:35.529 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:35.529 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:35.529 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:35.529 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.529 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:48:35.529 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:48:35.529 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:48:35.530 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:48:35.530 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:48:35.530 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:35.530 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:35.530 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.530 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:48:35.530 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:48:35.530 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.530 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:48:35.530 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:48:35.530 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:48:35.531 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:35.531 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:35.531 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:48:35.532 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.532 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:48:35.532 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:48:35.532 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 6 2023-11-16T01:48:35.532 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:35.532 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:35.532 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:48:35.532 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.532 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:48:35.532 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:48:35.532 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:48:35.533 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:35.533 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:35.533 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.533 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:48:35.533 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:48:35.534 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-16T01:48:35.534 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-11-16T01:48:35.534 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-16T01:48:35.671 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:48:35.672 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:48:35.672 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:48:35.672 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:48:35.672 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 12, 2023-11-16T01:48:35.672 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:48:35.672 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:48:35.672 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:48:35.672 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:35.672 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 0, 2023-11-16T01:48:35.672 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:48:35.672 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:48:35.673 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:48:35.673 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:48:35.673 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:48:35.673 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:48:35.673 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:48:35.673 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:48:35.673 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:48:35.673 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:48:35.673 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:48:35.673 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:35.673 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:48:35.674 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:48:35.674 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:48:35.674 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:48:35.674 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:48:35.674 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:48:35.674 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:48:35.674 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:48:35.674 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:48:35.674 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:48:35.674 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:35.674 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:48:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:48:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:48:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:48:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:48:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:48:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:48:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:48:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:48:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:48:35.675 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:48:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:48:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:48:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:48:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:48:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:48:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:48:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:48:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:48:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:48:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:48:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:48:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:48:35.676 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:48:35.677 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:48:35.677 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:35.677 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:48:35.677 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.677 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:48:35.677 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.677 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:48:35.677 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:48:35.677 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:48:35.677 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:48:35.678 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.678 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:48:35.678 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:48:35.678 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:35.678 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.678 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.678 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:48:35.678 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:48:35.678 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:35.678 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:35.678 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:48:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:48:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:48:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:48:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:48:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:48:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:48:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:48:35.679 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:48:35.680 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.680 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:48:35.680 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:48:35.680 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:35.680 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.680 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.680 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:48:35.680 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:48:35.680 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:35.680 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:35.681 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:35.681 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.681 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:48:35.682 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:48:35.682 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:48:35.682 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:48:35.682 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:48:35.682 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.682 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.682 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:48:35.682 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:48:35.682 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:48:35.682 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:48:35.683 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.683 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:48:35.683 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:48:35.683 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:35.683 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.683 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.683 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:48:35.683 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:48:35.683 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:48:35.683 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:35.683 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:35.683 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.684 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:48:35.684 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:48:35.684 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:48:35.684 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:48:35.684 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:48:35.684 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:35.684 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:35.684 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.684 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:48:35.684 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:48:35.685 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.685 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:48:35.685 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:48:35.685 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:48:35.685 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:35.685 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:35.685 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:48:35.685 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.685 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:48:35.685 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:48:35.686 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 3 2023-11-16T01:48:35.686 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:35.686 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:48:35.686 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": [ 2023-11-16T01:48:35.686 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:48:35.686 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:48:35.686 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:48:35.686 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:48:35.686 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:48:35.686 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:48:35.686 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:48:35.687 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:48:35.687 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:48:35.689 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-16T01:48:35.689 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-11-16T01:48:55.691 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-11-16T01:48:55.691 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-11-16T01:48:55.691 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-16T01:48:55.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-16T01:48:55.697 INFO:tasks.ceph.mon.b:Started 2023-11-16T01:48:55.698 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-16T01:48:55.699 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:49:01.058 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":16,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:49:04.070 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:49:04.403 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:49:04.417 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-16T01:49:04.417 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-16T01:49:04.417 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-16T01:49:04.556 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:49:04.556 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:49:04.556 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:49:04.557 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "leader", 2023-11-16T01:49:04.557 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 18, 2023-11-16T01:49:04.557 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:49:04.557 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:49:04.557 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:49:04.557 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:49:04.557 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:04.557 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 3, 2023-11-16T01:49:04.557 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:04.557 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:49:04.557 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:49:04.557 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:04.558 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:04.558 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:04.558 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:04.558 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:04.558 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:04.558 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:04.558 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:04.558 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:04.558 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:49:04.558 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:49:04.558 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:04.559 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:04.559 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:04.559 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:04.559 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:04.559 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:04.559 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:04.559 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:04.559 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:04.559 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.559 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:49:04.559 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:49:04.559 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:49:04.559 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:49:04.560 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:49:04.560 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:49:04.560 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:04.560 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:04.560 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:49:04.560 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:49:04.560 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:49:04.560 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:49:04.560 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:49:04.560 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:49:04.560 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:49:04.560 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:04.561 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:49:04.561 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:04.561 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:04.561 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:04.561 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:04.561 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:04.561 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:04.561 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:04.561 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:04.561 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:04.561 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:49:04.561 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.561 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:49:04.562 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.562 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:49:04.562 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:49:04.562 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:04.563 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:04.563 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.563 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:04.564 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:49:04.564 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:04.564 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.564 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.564 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:04.564 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:49:04.564 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:04.564 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:04.564 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:04.564 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.564 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:49:04.564 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:49:04.565 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:04.565 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:04.565 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:04.565 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.565 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.565 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:49:04.565 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:49:04.565 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:04.565 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:04.565 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.566 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:04.566 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:49:04.566 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:04.566 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.566 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.566 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:04.567 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:49:04.567 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:04.567 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:04.567 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:04.567 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.567 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:49:04.567 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:49:04.567 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:04.567 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:04.567 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:04.567 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.567 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.567 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:49:04.568 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:49:04.568 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:04.568 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:04.568 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.568 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:04.568 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:49:04.568 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:04.568 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.568 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.568 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:04.568 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:49:04.569 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:04.569 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:04.569 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:04.569 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.569 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:49:04.569 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:49:04.570 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:04.570 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:04.570 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:04.570 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:04.570 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:04.570 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.570 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:49:04.570 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:49:04.570 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.570 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:04.570 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:04.570 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:49:04.571 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:04.571 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:04.571 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:49:04.571 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.571 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:04.571 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:04.571 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 6 2023-11-16T01:49:04.571 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:04.571 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:04.571 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:49:04.571 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.571 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:04.571 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:04.572 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:49:04.572 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:04.572 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:04.572 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.572 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:49:04.572 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:49:04.577 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:49:04.578 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-16T01:49:04.578 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-16T01:49:04.725 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:49:04.725 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:49:04.726 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:49:04.726 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:49:04.726 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 18, 2023-11-16T01:49:04.726 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:49:04.726 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:49:04.726 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:49:04.726 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:49:04.726 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:04.726 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 3, 2023-11-16T01:49:04.726 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:04.726 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:49:04.726 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:49:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:49:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:49:04.727 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:04.728 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:04.728 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:04.728 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:04.728 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:04.728 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:04.728 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:04.728 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:04.728 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:04.728 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.728 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:49:04.728 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:49:04.728 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:49:04.729 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:49:04.729 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:49:04.729 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:49:04.729 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:04.729 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:04.729 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:49:04.729 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:49:04.729 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:49:04.729 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:49:04.729 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:49:04.729 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:49:04.730 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:49:04.730 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:04.730 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:49:04.730 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:04.730 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:04.730 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:04.731 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:04.731 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:04.731 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:04.731 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:04.731 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:04.732 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:04.732 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:49:04.732 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.732 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:49:04.732 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.732 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:49:04.732 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:49:04.732 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:04.732 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:04.732 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.733 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:04.733 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:49:04.733 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:04.733 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.733 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.735 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:04.735 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:49:04.735 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:04.735 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:04.735 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:04.735 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.735 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:49:04.735 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:49:04.736 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:04.736 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:04.736 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:04.736 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.736 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.736 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:49:04.736 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:49:04.736 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:04.736 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:04.736 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.736 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:04.736 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:49:04.737 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:04.737 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.737 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.737 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:04.737 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:49:04.737 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:04.737 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:04.737 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:04.737 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.737 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:49:04.737 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:49:04.737 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:04.738 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:04.738 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:04.738 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.738 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.738 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:49:04.738 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:49:04.738 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:04.738 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:04.738 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.739 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:04.739 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:49:04.739 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:04.739 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.739 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.739 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:04.739 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:49:04.739 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:04.739 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:04.739 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:04.739 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.740 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:49:04.740 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:49:04.740 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:04.740 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:04.740 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:04.740 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:04.740 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:04.740 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.741 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:49:04.741 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:49:04.748 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.748 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:04.748 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:04.749 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:49:04.749 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:04.750 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:04.750 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:49:04.750 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.750 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:04.750 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:04.750 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 3 2023-11-16T01:49:04.750 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:04.750 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:04.750 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": [ 2023-11-16T01:49:04.750 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:04.750 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:04.750 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:04.751 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:49:04.751 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:04.751 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:04.751 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:04.751 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:49:04.751 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:49:04.752 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:49:04.752 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-16T01:49:04.752 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-16T01:49:06.316 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:49:06.316 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:49:06.316 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:49:06.316 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:49:06.316 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 18, 2023-11-16T01:49:06.316 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:49:06.316 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:49:06.317 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:49:06.317 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:49:06.317 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:06.317 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 4, 2023-11-16T01:49:06.317 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:06.317 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:49:06.317 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:49:06.317 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:06.317 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:06.317 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:06.317 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:06.317 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:06.318 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:06.318 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:06.318 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:06.318 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:06.318 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:49:06.318 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:49:06.318 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:06.318 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:06.318 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:06.318 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:06.318 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:06.319 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:06.319 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:06.319 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:06.319 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:06.319 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:06.319 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:49:06.319 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:49:06.319 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:49:06.319 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:49:06.319 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:49:06.319 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:49:06.319 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:06.320 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:06.320 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:49:06.320 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:49:06.320 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:49:06.320 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:49:06.320 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:49:06.320 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:49:06.320 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:49:06.320 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:06.320 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:49:06.320 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:06.320 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:06.321 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:06.321 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:06.323 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:06.323 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:06.323 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:06.323 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:06.324 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:06.324 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:49:06.324 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:06.324 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:49:06.325 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:06.325 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:49:06.325 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:49:06.325 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:06.325 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:06.326 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:06.326 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:06.326 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:49:06.326 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:06.326 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:06.326 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:06.326 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:06.326 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:49:06.326 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:06.326 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:06.326 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:06.326 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:06.327 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:49:06.327 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:49:06.327 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:06.327 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:06.327 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:06.327 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:06.327 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:06.327 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:49:06.328 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:49:06.328 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:06.328 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:06.328 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:06.328 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:06.328 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:49:06.328 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:06.328 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:06.328 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:06.328 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:06.329 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:49:06.329 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:06.329 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:06.329 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:06.329 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:06.329 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:49:06.329 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:49:06.329 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:06.329 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:06.329 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:06.329 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:06.329 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:06.330 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:49:06.330 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:49:06.330 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:06.330 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:06.330 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:06.330 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:06.330 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:49:06.330 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:06.330 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:06.330 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:06.330 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:06.330 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:49:06.331 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:06.331 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:06.331 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:06.331 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:06.331 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:49:06.331 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:49:06.331 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:06.331 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:06.331 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:06.331 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:06.331 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:06.331 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:06.332 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:49:06.332 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:49:06.332 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:06.332 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:06.332 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:06.332 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:49:06.332 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:06.332 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:06.332 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:49:06.332 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:06.332 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:06.349 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:06.349 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:49:06.349 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:06.349 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:06.349 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:06.350 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:49:06.350 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:49:06.351 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:49:06.351 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-11-16T01:49:06.351 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-11-16T01:49:06.708 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:49:06.706+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:49:06.708 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:49:06.706+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=12,kv=6,logm=68} crc {auth=2727138964,config=3140514009,health=3806841325,kv=3795632867,logm=560614510}) 2023-11-16T01:49:06.708 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:49:06.706+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=11,kv=6,logm=70} crc {auth=3300275799,config=3140514009,health=3035693388,kv=2868953028,logm=2611536142}) 2023-11-16T01:49:06.708 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:49:06.706+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:49:06.709 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:49:06.706+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=12,kv=6,logm=68} crc {auth=2727138964,config=3140514009,health=3806841325,kv=3795632867,logm=560614510}) 2023-11-16T01:49:06.709 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:49:06.706+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=5,logm=69,mdsmap=1} crc {auth=2727138964,config=3140514009,health=3035693388,kv=2986745882,logm=1750421395,mdsmap=2826658659}) 2023-11-16T01:49:06.715 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:49:06.714+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:49:06.715 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:49:06.714+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=2,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=33} crc {logm=469556867,mdsmap=1531151339,mgr=3338689344,mgr_command_descs=1448462394,mgr_metadata=2889854584,mgrstat=2362423902,mon_config_key=1707289356,monmap=1549841071,osd_metadata=2408539844,osd_pg_creating=534015559,osdmap=3574647424}) 2023-11-16T01:49:06.715 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:49:06.714+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=2,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=31} crc {logm=2691550706,mdsmap=1531151339,mgr=3338689344,mgr_command_descs=1448462394,mgr_metadata=2889854584,mgrstat=2362423902,mon_config_key=1707289356,monmap=1549841071,osd_metadata=2408539844,osd_pg_creating=534015559,osdmap=2514423803}) 2023-11-16T01:49:06.715 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:49:06.714+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:49:06.715 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:49:06.714+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=2,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=33} crc {logm=469556867,mdsmap=1531151339,mgr=3338689344,mgr_command_descs=1448462394,mgr_metadata=2889854584,mgrstat=2362423902,mon_config_key=1707289356,monmap=1549841071,osd_metadata=2408539844,osd_pg_creating=534015559,osdmap=3574647424}) 2023-11-16T01:49:06.715 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:49:06.714+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=2,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=30} crc {logm=469556867,mdsmap=1531151339,mgr=3338689344,mgr_command_descs=1448462394,mgr_metadata=2889854584,mgrstat=2362423902,mon_config_key=1707289356,monmap=1549841071,osd_metadata=2408539844,osd_pg_creating=534015559,osdmap=3096776731}) 2023-11-16T01:49:06.723 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-11-16T01:49:07.723 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-16T01:49:07.724 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:49:08.066 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:49:08.082 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-16T01:49:08.082 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-11-16T01:49:08.082 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-16T01:49:08.082 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-16T01:49:08.219 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:49:08.219 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:49:08.219 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:49:08.219 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "leader", 2023-11-16T01:49:08.221 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 18, 2023-11-16T01:49:08.221 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:49:08.221 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:49:08.221 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:49:08.221 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:49:08.221 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:08.221 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 6, 2023-11-16T01:49:08.221 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:08.222 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:49:08.222 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:49:08.222 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:08.222 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:08.222 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:08.222 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:08.222 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:08.222 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:08.222 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:08.222 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:08.222 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:08.223 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:49:08.223 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:49:08.223 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:08.223 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:08.223 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:08.223 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:08.223 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:08.223 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:08.223 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:08.223 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:08.223 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.223 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.224 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:49:08.224 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:49:08.224 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:49:08.224 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:49:08.224 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:49:08.224 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:49:08.224 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:08.224 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:08.224 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:49:08.224 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:49:08.224 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:49:08.224 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:49:08.224 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:49:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:49:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:49:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:49:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:08.226 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:08.226 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:49:08.226 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.226 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:49:08.226 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.226 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:49:08.226 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:49:08.226 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:08.226 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:08.226 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.226 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:08.227 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:49:08.227 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.227 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.227 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.227 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:08.227 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:49:08.227 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.227 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.227 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.227 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.227 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:49:08.227 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:49:08.228 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:08.228 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:08.228 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:08.228 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.228 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.228 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:49:08.228 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:49:08.228 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:08.228 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:08.228 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.228 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:08.228 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:49:08.229 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.229 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.229 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.229 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:08.229 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:49:08.229 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.229 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.229 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.229 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.229 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:49:08.229 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:49:08.229 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:08.230 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:08.230 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:08.230 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.230 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.230 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:49:08.230 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:49:08.230 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:08.230 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:08.230 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.230 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:08.230 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:49:08.231 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.231 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.231 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.231 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:08.231 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:49:08.231 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.231 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.231 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.231 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.231 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:49:08.231 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:49:08.231 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:08.232 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:08.232 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:08.232 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.232 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.232 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.232 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:49:08.232 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:49:08.232 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.232 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:08.232 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:08.232 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:49:08.232 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.233 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:08.233 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:49:08.233 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.233 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:08.233 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:08.233 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 6 2023-11-16T01:49:08.233 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.233 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:08.233 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:49:08.233 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.233 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:08.233 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:08.234 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:49:08.234 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.234 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.234 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.234 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:49:08.234 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:49:08.239 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:49:08.240 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-16T01:49:08.240 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-16T01:49:08.379 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:49:08.379 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:49:08.380 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:49:08.380 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:49:08.380 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 18, 2023-11-16T01:49:08.380 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:49:08.380 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:49:08.380 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:49:08.380 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:49:08.380 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:08.380 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 6, 2023-11-16T01:49:08.380 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:08.380 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:49:08.380 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:49:08.381 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:08.381 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:08.381 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:08.381 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:08.381 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:08.381 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:08.381 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:08.381 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:08.381 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:08.381 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:49:08.381 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:49:08.381 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:08.382 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:08.382 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:08.382 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:08.382 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:08.382 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:08.382 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:08.382 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:08.382 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.382 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.382 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:49:08.382 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:49:08.382 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:49:08.383 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:49:08.383 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:49:08.383 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:49:08.383 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:08.383 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:08.383 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:49:08.384 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:49:08.384 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:49:08.384 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:49:08.384 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:49:08.385 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:49:08.385 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:49:08.385 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:08.385 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:49:08.385 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:08.385 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:08.385 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:08.386 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:08.386 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:08.386 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:08.386 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:08.386 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:08.386 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:08.386 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:49:08.386 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.386 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:49:08.387 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.387 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:49:08.387 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:49:08.387 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:08.387 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:08.387 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.387 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:08.387 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:49:08.388 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.388 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.388 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.388 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:08.388 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:49:08.388 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.389 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.389 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.389 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.389 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:49:08.389 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:49:08.389 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:08.389 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:08.389 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:08.389 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.389 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.392 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:49:08.392 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:49:08.392 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:08.392 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:08.392 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.392 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:08.392 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:49:08.393 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.393 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.393 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.393 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:08.393 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:49:08.393 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.393 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.394 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.394 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.394 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:49:08.395 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:49:08.395 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:08.395 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:08.395 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:08.395 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.395 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.396 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:49:08.396 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:49:08.396 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:08.396 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:08.396 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.398 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:08.398 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:49:08.398 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.400 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.401 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.401 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:08.401 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:49:08.401 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.401 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.401 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.401 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.401 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:49:08.401 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:49:08.401 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:08.401 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:08.402 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:08.402 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.402 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.402 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.402 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:49:08.402 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:49:08.402 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.402 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:08.402 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:08.402 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:49:08.402 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.403 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:08.403 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:49:08.403 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.403 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:08.403 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:08.403 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 3 2023-11-16T01:49:08.403 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.403 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:08.403 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": [ 2023-11-16T01:49:08.403 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.403 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:08.403 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:08.404 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:49:08.404 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.404 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.404 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.404 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:49:08.404 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:49:08.406 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:49:08.406 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-16T01:49:08.407 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-16T01:49:08.547 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:49:08.547 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:49:08.547 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:49:08.547 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:49:08.547 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 18, 2023-11-16T01:49:08.547 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:49:08.547 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:49:08.547 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:49:08.547 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:49:08.548 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:08.548 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 7, 2023-11-16T01:49:08.548 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:08.548 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:49:08.548 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:49:08.548 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:08.548 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:08.548 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:08.548 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:08.548 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:08.548 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:08.548 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:08.549 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:08.549 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:08.549 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:49:08.549 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:49:08.549 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:08.549 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:08.549 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:08.549 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:08.549 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:08.549 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:08.549 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:08.550 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:08.550 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.550 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.550 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:49:08.550 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:49:08.550 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:49:08.550 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:49:08.550 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:49:08.550 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:49:08.550 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:08.550 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:08.551 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:49:08.551 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:49:08.551 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:49:08.551 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:49:08.551 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:49:08.551 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:49:08.551 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:49:08.551 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:08.551 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:49:08.551 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:08.551 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:08.551 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:08.552 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:08.552 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:08.552 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:08.552 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:08.552 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:08.552 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:08.552 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:49:08.552 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.552 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:49:08.552 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.552 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:49:08.552 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:49:08.553 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:08.553 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:08.553 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.553 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:08.553 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:49:08.553 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.553 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.553 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.553 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:08.553 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:49:08.554 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.554 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.554 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.554 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.554 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:49:08.554 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:49:08.554 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:08.554 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:08.554 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:08.554 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.554 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.555 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:49:08.555 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:49:08.555 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:08.556 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:08.556 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.556 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:08.556 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:49:08.557 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.557 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.557 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.557 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:08.557 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:49:08.557 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.557 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.557 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.557 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.558 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:49:08.558 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:49:08.558 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:08.558 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:08.558 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:08.558 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.558 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.558 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:49:08.558 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:49:08.558 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:08.559 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:08.559 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.559 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:08.559 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:49:08.559 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.560 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.560 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.560 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:08.560 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:49:08.560 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:08.560 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.560 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.561 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.561 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:49:08.561 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:49:08.561 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:08.561 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:08.561 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:08.561 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.561 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.561 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.561 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:49:08.562 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:49:08.562 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.562 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:08.562 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:08.562 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:49:08.562 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.562 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:08.562 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:49:08.563 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:08.563 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:08.563 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:08.563 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:49:08.563 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:08.563 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:08.563 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:08.564 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:49:08.564 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:49:08.565 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:49:08.565 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-11-16T01:49:08.565 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-11-16T01:49:08.566 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-11-16T01:49:08.566 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-11-16T01:49:08.566 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-16T01:49:08.566 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T01:49:08.579 INFO:tasks.ceph.mon.b:Stopped 2023-11-16T01:49:08.579 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-11-16T01:49:08.580 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:49:08.744 INFO:tasks.workunit.client.0.smithi136.stderr:Note: switching to 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac'. 2023-11-16T01:49:08.745 INFO:tasks.workunit.client.0.smithi136.stderr: 2023-11-16T01:49:08.745 INFO:tasks.workunit.client.0.smithi136.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-11-16T01:49:08.745 INFO:tasks.workunit.client.0.smithi136.stderr:changes and commit them, and you can discard any commits you make in this 2023-11-16T01:49:08.745 INFO:tasks.workunit.client.0.smithi136.stderr:state without impacting any branches by switching back to a branch. 2023-11-16T01:49:08.745 INFO:tasks.workunit.client.0.smithi136.stderr: 2023-11-16T01:49:08.745 INFO:tasks.workunit.client.0.smithi136.stderr:If you want to create a new branch to retain commits you create, you may 2023-11-16T01:49:08.745 INFO:tasks.workunit.client.0.smithi136.stderr:do so (now or later) by using -c with the switch command. Example: 2023-11-16T01:49:08.745 INFO:tasks.workunit.client.0.smithi136.stderr: 2023-11-16T01:49:08.745 INFO:tasks.workunit.client.0.smithi136.stderr: git switch -c 2023-11-16T01:49:08.745 INFO:tasks.workunit.client.0.smithi136.stderr: 2023-11-16T01:49:08.746 INFO:tasks.workunit.client.0.smithi136.stderr:Or undo this operation with: 2023-11-16T01:49:08.746 INFO:tasks.workunit.client.0.smithi136.stderr: 2023-11-16T01:49:08.746 INFO:tasks.workunit.client.0.smithi136.stderr: git switch - 2023-11-16T01:49:08.746 INFO:tasks.workunit.client.0.smithi136.stderr: 2023-11-16T01:49:08.746 INFO:tasks.workunit.client.0.smithi136.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-11-16T01:49:08.746 INFO:tasks.workunit.client.0.smithi136.stderr: 2023-11-16T01:49:08.746 INFO:tasks.workunit.client.0.smithi136.stderr:HEAD is now at ed3c8e926b9 Merge branch 'wip-63441-pacific' of https://github.com/k0ste/ceph into wip-yuri3-testing-2023-11-14-1227-pacific 2023-11-16T01:49:08.752 DEBUG:teuthology.orchestra.run.smithi136:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-11-16T01:49:08.773 INFO:tasks.workunit.client.0.smithi136.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-11-16T01:49:08.776 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-16T01:49:08.777 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-11-16T01:49:08.921 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:49:09.643 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-11-16T01:49:09.721 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-11-16T01:49:09.770 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-16T01:49:09.771 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-16T01:49:09.772 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-11-16T01:49:09.824 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-16T01:49:09.829 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-11-16T01:49:09.829 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-11-16T01:49:09.836 INFO:tasks.workunit:Running workunits matching mon/pool_ops.sh on client.0... 2023-11-16T01:49:09.837 INFO:tasks.workunit:Running workunit mon/pool_ops.sh... 2023-11-16T01:49:09.837 DEBUG:teuthology.orchestra.run.smithi136:workunit test mon/pool_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ed3c8e926b9de93026c8a7c1d86be5d6821b52ac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pool_ops.sh 2023-11-16T01:49:09.893 INFO:tasks.workunit.client.0.smithi136.stderr:+ TEST_POOL=testpool1234 2023-11-16T01:49:09.894 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool create testpool1234 8 --autoscale-mode off 2023-11-16T01:49:11.938 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:49:12.289 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:49:15.303 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:49:15.646 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:49:18.660 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:49:24.554 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'testpool1234' already exists 2023-11-16T01:49:34.073 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set testpool1234 pg_num_min 2 2023-11-16T01:49:34.124 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 3 pg_num_min to 2 2023-11-16T01:49:34.125 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool get testpool1234 pg_num_min 2023-11-16T01:49:34.126 INFO:tasks.workunit.client.0.smithi136.stderr:+ grep 2 2023-11-16T01:49:34.126 INFO:tasks.workunit.client.0.smithi136.stdout:pg_num_min: 2 2023-11-16T01:49:34.127 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":22,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:49:34.127 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-11-16T01:49:34.127 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-11-16T01:49:34.128 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-16T01:49:34.128 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set testpool1234 pg_num_max 33 2023-11-16T01:49:34.143 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 3 pg_num_max to 33 2023-11-16T01:49:34.143 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool get testpool1234 pg_num_max 2023-11-16T01:49:34.143 INFO:tasks.workunit.client.0.smithi136.stderr:+ grep 33 2023-11-16T01:49:34.144 INFO:tasks.workunit.client.0.smithi136.stdout:pg_num_max: 33 2023-11-16T01:49:34.147 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_min 9 2023-11-16T01:49:34.147 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:49:34.147 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set testpool1234 pg_num_min 9 2023-11-16T01:49:34.147 INFO:tasks.workunit.client.0.smithi136.stderr:Error EINVAL: specified pg_num_min 9 > pg_num 8 2023-11-16T01:49:34.147 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:49:34.147 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_max 7 2023-11-16T01:49:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:49:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set testpool1234 pg_num_max 7 2023-11-16T01:49:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:Error EINVAL: specified pg_num_max 7 < pg_num 8 2023-11-16T01:49:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:49:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 1 2023-11-16T01:49:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:49:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set testpool1234 pg_num 1 2023-11-16T01:49:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:Error EINVAL: specified pg_num 1 < pg_num_min 2 2023-11-16T01:49:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:49:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 44 2023-11-16T01:49:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:49:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set testpool1234 pg_num 44 2023-11-16T01:49:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:Error EINVAL: specified pg_num 44 < pg_num_max 33 2023-11-16T01:49:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:49:34.149 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set testpool1234 pg_num_min 0 2023-11-16T01:49:34.149 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 3 pg_num_min to 0 2023-11-16T01:49:34.149 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_min 2023-11-16T01:49:34.149 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:49:34.149 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool get testpool1234 pg_num_min 2023-11-16T01:49:34.149 INFO:tasks.workunit.client.0.smithi136.stderr:Error ENOENT: option 'pg_num_min' is not set on pool 'testpool1234' 2023-11-16T01:49:34.149 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:49:34.149 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set testpool1234 pg_num_max 0 2023-11-16T01:49:34.149 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 3 pg_num_max to 0 2023-11-16T01:49:34.149 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_max 2023-11-16T01:49:34.149 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:49:34.149 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool get testpool1234 pg_num_max 2023-11-16T01:49:34.292 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:49:34.293 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:49:34.293 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:49:34.293 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "leader", 2023-11-16T01:49:34.293 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 22, 2023-11-16T01:49:34.293 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:49:34.293 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:49:34.294 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:49:34.294 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:34.294 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 10, 2023-11-16T01:49:34.294 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:34.294 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:49:34.294 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:49:34.294 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:34.294 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:34.294 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:34.295 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:34.295 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:34.295 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:34.295 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:34.295 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:34.295 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:34.295 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:49:34.295 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:49:34.295 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:34.295 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:34.295 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:34.296 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:34.296 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:34.296 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:34.296 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:34.296 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:34.296 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:34.296 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.296 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:49:34.296 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:49:34.296 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:49:34.296 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:49:34.296 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:49:34.296 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:49:34.297 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:34.297 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:34.297 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:49:34.297 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:49:34.297 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:49:34.297 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:49:34.297 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:49:34.297 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:49:34.297 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:49:34.297 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:34.297 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:49:34.297 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:34.297 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:34.298 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:34.298 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:34.298 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:34.298 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:34.298 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:34.298 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:34.298 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:34.298 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:49:34.298 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.298 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:49:34.298 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.298 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:49:34.299 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:49:34.299 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:34.299 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:34.299 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.299 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:34.299 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:49:34.299 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:34.299 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.299 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.299 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:34.299 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:49:34.299 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:34.299 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:34.299 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:34.300 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.300 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:49:34.300 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:49:34.300 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:34.300 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:34.300 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:34.300 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.300 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.300 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:49:34.300 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:49:34.300 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:34.300 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:34.301 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.301 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:34.301 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:49:34.301 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:34.301 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.301 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.301 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:34.301 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:49:34.301 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:49:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:49:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:34.305 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:34.305 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.306 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.306 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:49:34.306 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:49:34.306 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:34.306 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:34.306 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.306 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:34.306 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:49:34.306 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:34.306 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.306 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.306 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:34.355 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:49:34.355 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:34.356 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:34.356 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:34.356 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.356 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:49:34.356 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:49:34.356 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:34.356 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:34.356 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:34.356 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:49:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:49:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:49:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:34.357 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:49:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 6 2023-11-16T01:49:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:49:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:34.358 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:49:34.359 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:34.359 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:34.359 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.359 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:49:34.359 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:49:34.365 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-16T01:49:34.365 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-11-16T01:49:34.366 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-16T01:49:34.512 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:49:34.512 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:49:34.512 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:49:34.512 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:49:34.512 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 22, 2023-11-16T01:49:34.513 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:49:34.513 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:49:34.513 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:49:34.513 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:34.513 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 11, 2023-11-16T01:49:34.513 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:34.513 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:49:34.513 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:49:34.513 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:34.513 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:34.513 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:34.513 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:34.513 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:34.514 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:34.514 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:34.514 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:34.514 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:34.514 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:49:34.514 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:49:34.514 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:34.514 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:34.514 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:34.515 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:34.515 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:34.515 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:34.515 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:34.515 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:34.515 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:34.515 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.515 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:49:34.515 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:49:34.515 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:49:34.515 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:49:34.515 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:49:34.516 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:49:34.516 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:34.516 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:49:34.516 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:49:34.516 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:49:34.516 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:49:34.516 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:49:34.516 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:49:34.516 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:49:34.516 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:49:34.516 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:49:34.516 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:49:34.516 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:49:34.517 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:49:34.517 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:49:34.517 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:49:34.517 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:49:34.517 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:49:34.517 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:49:34.517 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:49:34.517 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:34.517 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:49:34.517 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.517 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:49:34.517 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.517 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:49:34.518 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:49:34.518 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:34.518 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:34.518 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.518 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:34.518 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:49:34.518 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:34.520 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.520 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.520 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:34.520 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:49:34.520 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:34.520 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:34.520 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:34.520 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.520 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:49:34.521 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:49:34.521 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:34.521 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:34.521 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:34.521 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.521 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.521 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:49:34.521 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:49:34.521 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:34.522 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:34.522 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.522 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:34.522 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:49:34.522 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:34.522 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.522 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.522 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:34.522 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:49:34.522 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:34.523 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:34.523 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:34.523 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.523 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:49:34.523 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:49:34.523 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:34.523 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:34.523 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:34.523 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.523 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.523 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:49:34.524 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:49:34.524 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:49:34.524 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:49:34.524 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.524 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:49:34.524 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:49:34.524 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:34.524 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.524 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.524 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:49:34.524 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:49:34.525 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:49:34.525 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:34.525 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:34.525 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.525 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:49:34.525 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:49:34.525 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:49:34.525 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:49:34.525 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:49:34.525 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:34.525 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:34.526 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.526 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:49:34.526 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:49:34.526 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.526 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:34.526 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:34.526 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:49:34.526 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:34.527 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:34.527 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:49:34.527 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.527 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:34.527 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:34.527 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 4 2023-11-16T01:49:34.527 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:34.527 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:49:34.528 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": [ 2023-11-16T01:49:34.528 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:49:34.528 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:49:34.528 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:49:34.528 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:49:34.528 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:49:34.528 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:49:34.528 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:49:34.528 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:49:34.529 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:49:34.530 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-16T01:49:34.530 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-11-16T01:49:34.670 INFO:tasks.workunit.client.0.smithi136.stderr:Error ENOENT: option 'pg_num_max' is not set on pool 'testpool1234' 2023-11-16T01:49:34.671 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:49:34.672 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool delete testpool1234 testpool1234 --yes-i-really-really-mean-it 2023-11-16T01:49:35.662 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'testpool1234' does not exist 2023-11-16T01:49:35.672 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false ceph osd pool create foo 123 123 invalid foo-profile foo-ruleset 2023-11-16T01:49:35.672 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:49:35.673 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool create foo 123 123 invalid foo-profile foo-ruleset 2023-11-16T01:49:35.959 INFO:tasks.workunit.client.0.smithi136.stderr:foo-ruleset not valid: foo-ruleset doesn't represent a float 2023-11-16T01:49:35.959 INFO:tasks.workunit.client.0.smithi136.stderr:Invalid command: unused arguments: ['foo-ruleset'] 2023-11-16T01:49:35.959 INFO:tasks.workunit.client.0.smithi136.stderr:osd pool create [] [] [replicated|erasure] [] [] [] [] [] [] [on|off|warn] [--bulk] [] [] : create pool 2023-11-16T01:49:35.959 INFO:tasks.workunit.client.0.smithi136.stderr:Error EINVAL: invalid command 2023-11-16T01:49:35.962 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:49:35.962 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool create foo 123 123 replicated 2023-11-16T01:49:36.674 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'foo' already exists 2023-11-16T01:49:36.683 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool create fooo 123 123 erasure default 2023-11-16T01:49:39.529 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'fooo' already exists 2023-11-16T01:49:39.537 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool create foooo 123 2023-11-16T01:49:40.723 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'foooo' already exists 2023-11-16T01:49:40.732 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool create foo 123 2023-11-16T01:49:41.111 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'foo' already exists 2023-11-16T01:49:41.120 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set foo size 1 --yes-i-really-mean-it 2023-11-16T01:49:42.644 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 4 size to 1 2023-11-16T01:49:42.657 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_config_value foo min_size 1 2023-11-16T01:49:42.658 INFO:tasks.workunit.client.0.smithi136.stderr:+ local pool_name config_opt expected_val val 2023-11-16T01:49:42.658 INFO:tasks.workunit.client.0.smithi136.stderr:+ pool_name=foo 2023-11-16T01:49:42.658 INFO:tasks.workunit.client.0.smithi136.stderr:+ config_opt=min_size 2023-11-16T01:49:42.658 INFO:tasks.workunit.client.0.smithi136.stderr:+ expected_val=1 2023-11-16T01:49:42.658 INFO:tasks.workunit.client.0.smithi136.stderr:++ get_config_value_or_die foo min_size 2023-11-16T01:49:42.658 INFO:tasks.workunit.client.0.smithi136.stderr:++ local pool_name config_opt raw val 2023-11-16T01:49:42.658 INFO:tasks.workunit.client.0.smithi136.stderr:++ pool_name=foo 2023-11-16T01:49:42.658 INFO:tasks.workunit.client.0.smithi136.stderr:++ config_opt=min_size 2023-11-16T01:49:42.658 INFO:tasks.workunit.client.0.smithi136.stderr:+++ ceph osd pool get foo min_size 2023-11-16T01:49:43.023 INFO:tasks.workunit.client.0.smithi136.stderr:++ raw='min_size: 1' 2023-11-16T01:49:43.024 INFO:tasks.workunit.client.0.smithi136.stderr:++ [[ 0 -ne 0 ]] 2023-11-16T01:49:43.024 INFO:tasks.workunit.client.0.smithi136.stderr:+++ echo min_size: 1 2023-11-16T01:49:43.024 INFO:tasks.workunit.client.0.smithi136.stderr:+++ sed -e 's/[{} "]//g' 2023-11-16T01:49:43.026 INFO:tasks.workunit.client.0.smithi136.stderr:++ raw=min_size:1 2023-11-16T01:49:43.027 INFO:tasks.workunit.client.0.smithi136.stderr:+++ echo min_size:1 2023-11-16T01:49:43.027 INFO:tasks.workunit.client.0.smithi136.stderr:+++ cut -f2 -d: 2023-11-16T01:49:43.028 INFO:tasks.workunit.client.0.smithi136.stderr:++ val=1 2023-11-16T01:49:43.028 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo 1 2023-11-16T01:49:43.028 INFO:tasks.workunit.client.0.smithi136.stderr:++ return 0 2023-11-16T01:49:43.029 INFO:tasks.workunit.client.0.smithi136.stderr:+ val=1 2023-11-16T01:49:43.029 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ 1 != \1 ]] 2023-11-16T01:49:43.029 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set foo size 4 2023-11-16T01:49:45.178 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 4 size to 4 2023-11-16T01:49:45.200 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_config_value foo min_size 2 2023-11-16T01:49:45.200 INFO:tasks.workunit.client.0.smithi136.stderr:+ local pool_name config_opt expected_val val 2023-11-16T01:49:45.200 INFO:tasks.workunit.client.0.smithi136.stderr:+ pool_name=foo 2023-11-16T01:49:45.200 INFO:tasks.workunit.client.0.smithi136.stderr:+ config_opt=min_size 2023-11-16T01:49:45.200 INFO:tasks.workunit.client.0.smithi136.stderr:+ expected_val=2 2023-11-16T01:49:45.202 INFO:tasks.workunit.client.0.smithi136.stderr:++ get_config_value_or_die foo min_size 2023-11-16T01:49:45.202 INFO:tasks.workunit.client.0.smithi136.stderr:++ local pool_name config_opt raw val 2023-11-16T01:49:45.202 INFO:tasks.workunit.client.0.smithi136.stderr:++ pool_name=foo 2023-11-16T01:49:45.202 INFO:tasks.workunit.client.0.smithi136.stderr:++ config_opt=min_size 2023-11-16T01:49:45.202 INFO:tasks.workunit.client.0.smithi136.stderr:+++ ceph osd pool get foo min_size 2023-11-16T01:49:45.591 INFO:tasks.workunit.client.0.smithi136.stderr:++ raw='min_size: 2' 2023-11-16T01:49:45.592 INFO:tasks.workunit.client.0.smithi136.stderr:++ [[ 0 -ne 0 ]] 2023-11-16T01:49:45.592 INFO:tasks.workunit.client.0.smithi136.stderr:+++ echo min_size: 2 2023-11-16T01:49:45.592 INFO:tasks.workunit.client.0.smithi136.stderr:+++ sed -e 's/[{} "]//g' 2023-11-16T01:49:45.595 INFO:tasks.workunit.client.0.smithi136.stderr:++ raw=min_size:2 2023-11-16T01:49:45.595 INFO:tasks.workunit.client.0.smithi136.stderr:+++ echo min_size:2 2023-11-16T01:49:45.595 INFO:tasks.workunit.client.0.smithi136.stderr:+++ cut -f2 -d: 2023-11-16T01:49:45.597 INFO:tasks.workunit.client.0.smithi136.stderr:++ val=2 2023-11-16T01:49:45.597 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo 2 2023-11-16T01:49:45.597 INFO:tasks.workunit.client.0.smithi136.stderr:++ return 0 2023-11-16T01:49:45.597 INFO:tasks.workunit.client.0.smithi136.stderr:+ val=2 2023-11-16T01:49:45.597 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ 2 != \2 ]] 2023-11-16T01:49:45.597 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set foo size 10 2023-11-16T01:49:47.294 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 4 size to 10 2023-11-16T01:49:47.308 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_config_value foo min_size 5 2023-11-16T01:49:47.308 INFO:tasks.workunit.client.0.smithi136.stderr:+ local pool_name config_opt expected_val val 2023-11-16T01:49:47.308 INFO:tasks.workunit.client.0.smithi136.stderr:+ pool_name=foo 2023-11-16T01:49:47.308 INFO:tasks.workunit.client.0.smithi136.stderr:+ config_opt=min_size 2023-11-16T01:49:47.308 INFO:tasks.workunit.client.0.smithi136.stderr:+ expected_val=5 2023-11-16T01:49:47.310 INFO:tasks.workunit.client.0.smithi136.stderr:++ get_config_value_or_die foo min_size 2023-11-16T01:49:47.310 INFO:tasks.workunit.client.0.smithi136.stderr:++ local pool_name config_opt raw val 2023-11-16T01:49:47.310 INFO:tasks.workunit.client.0.smithi136.stderr:++ pool_name=foo 2023-11-16T01:49:47.310 INFO:tasks.workunit.client.0.smithi136.stderr:++ config_opt=min_size 2023-11-16T01:49:47.310 INFO:tasks.workunit.client.0.smithi136.stderr:+++ ceph osd pool get foo min_size 2023-11-16T01:49:47.737 INFO:tasks.workunit.client.0.smithi136.stderr:++ raw='min_size: 5' 2023-11-16T01:49:47.737 INFO:tasks.workunit.client.0.smithi136.stderr:++ [[ 0 -ne 0 ]] 2023-11-16T01:49:47.737 INFO:tasks.workunit.client.0.smithi136.stderr:+++ echo min_size: 5 2023-11-16T01:49:47.737 INFO:tasks.workunit.client.0.smithi136.stderr:+++ sed -e 's/[{} "]//g' 2023-11-16T01:49:47.740 INFO:tasks.workunit.client.0.smithi136.stderr:++ raw=min_size:5 2023-11-16T01:49:47.740 INFO:tasks.workunit.client.0.smithi136.stderr:+++ echo min_size:5 2023-11-16T01:49:47.741 INFO:tasks.workunit.client.0.smithi136.stderr:+++ cut -f2 -d: 2023-11-16T01:49:47.742 INFO:tasks.workunit.client.0.smithi136.stderr:++ val=5 2023-11-16T01:49:47.743 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo 5 2023-11-16T01:49:47.743 INFO:tasks.workunit.client.0.smithi136.stderr:++ return 0 2023-11-16T01:49:47.743 INFO:tasks.workunit.client.0.smithi136.stderr:+ val=5 2023-11-16T01:49:47.743 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ 5 != \5 ]] 2023-11-16T01:49:47.743 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false ceph osd pool set foo size 0 2023-11-16T01:49:47.744 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:49:47.744 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set foo size 0 2023-11-16T01:49:48.030 INFO:tasks.workunit.client.0.smithi136.stderr:Error EINVAL: pool size must be between 1 and 10 2023-11-16T01:49:48.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:49:48.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false ceph osd pool set foo size 20 2023-11-16T01:49:48.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:49:48.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set foo size 20 2023-11-16T01:49:48.318 INFO:tasks.workunit.client.0.smithi136.stderr:Error EINVAL: pool size must be between 1 and 10 2023-11-16T01:49:48.319 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:49:48.319 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set foo size 3 2023-11-16T01:49:50.400 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 4 size to 3 2023-11-16T01:49:50.427 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd getcrushmap -o crush 2023-11-16T01:49:50.806 INFO:tasks.workunit.client.0.smithi136.stderr:5 2023-11-16T01:49:50.813 INFO:tasks.workunit.client.0.smithi136.stderr:+ crushtool -d crush -o crush.txt 2023-11-16T01:49:50.833 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed -i 's/max_size 10/max_size 3/' crush.txt 2023-11-16T01:49:50.836 INFO:tasks.workunit.client.0.smithi136.stderr:+ crushtool -c crush.txt -o crush.new 2023-11-16T01:49:50.855 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd setcrushmap -i crush.new 2023-11-16T01:49:52.416 INFO:tasks.workunit.client.0.smithi136.stderr:7 2023-11-16T01:49:52.426 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false ceph osd pool set foo size 4 2023-11-16T01:49:52.426 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:49:52.427 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set foo size 4 2023-11-16T01:49:52.718 INFO:tasks.workunit.client.0.smithi136.stderr:Error EINVAL: pool size is bigger than the crush rule max size 2023-11-16T01:49:52.721 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:49:52.722 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd setcrushmap -i crush 2023-11-16T01:49:54.531 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-11-16T01:49:54.531 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-11-16T01:49:54.531 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-16T01:49:54.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-16T01:49:54.535 INFO:tasks.ceph.mon.b:Started 2023-11-16T01:49:54.535 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-16T01:49:54.536 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:49:54.609 INFO:tasks.workunit.client.0.smithi136.stderr:9 2023-11-16T01:49:54.623 INFO:tasks.workunit.client.0.smithi136.stderr:+ rm -f crush crush.txt crush.new 2023-11-16T01:49:54.623 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false ceph osd pool delete foo 2023-11-16T01:49:54.623 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:49:54.623 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool delete foo 2023-11-16T01:49:54.878 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":22,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":31,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:49:54.898 INFO:tasks.workunit.client.0.smithi136.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2023-11-16T01:49:54.900 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:49:54.900 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false ceph osd pool delete foo foo 2023-11-16T01:49:54.900 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:49:54.900 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool delete foo foo 2023-11-16T01:49:57.892 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:50:00.136 INFO:tasks.workunit.client.0.smithi136.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2023-11-16T01:50:00.136 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:50:00.136 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false ceph osd pool delete foo foo --force 2023-11-16T01:50:00.136 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:50:00.137 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool delete foo foo --force 2023-11-16T01:50:00.416 INFO:tasks.workunit.client.0.smithi136.stderr:--force not valid: --force not one of 'true', 'false' 2023-11-16T01:50:00.417 INFO:tasks.workunit.client.0.smithi136.stderr:Invalid command: unused arguments: ['--force'] 2023-11-16T01:50:00.417 INFO:tasks.workunit.client.0.smithi136.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2023-11-16T01:50:00.417 INFO:tasks.workunit.client.0.smithi136.stderr:Error EINVAL: invalid command 2023-11-16T01:50:00.420 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:50:00.420 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false ceph osd pool delete foo fooo --yes-i-really-mean-it 2023-11-16T01:50:00.420 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:50:00.420 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool delete foo fooo --yes-i-really-mean-it 2023-11-16T01:50:00.694 INFO:tasks.workunit.client.0.smithi136.stderr:--yes-i-really-mean-it not valid: --yes-i-really-mean-it not one of 'true', 'false' 2023-11-16T01:50:00.694 INFO:tasks.workunit.client.0.smithi136.stderr:Invalid command: unused arguments: ['--yes-i-really-mean-it'] 2023-11-16T01:50:00.694 INFO:tasks.workunit.client.0.smithi136.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2023-11-16T01:50:00.694 INFO:tasks.workunit.client.0.smithi136.stderr:Error EINVAL: invalid command 2023-11-16T01:50:00.697 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:50:00.697 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false ceph osd pool delete foo --yes-i-really-mean-it foo 2023-11-16T01:50:00.697 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:50:00.697 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool delete foo --yes-i-really-mean-it foo 2023-11-16T01:50:00.984 INFO:tasks.workunit.client.0.smithi136.stderr:foo not valid: foo not one of 'true', 'false' 2023-11-16T01:50:00.984 INFO:tasks.workunit.client.0.smithi136.stderr:Invalid command: unused arguments: ['foo'] 2023-11-16T01:50:00.984 INFO:tasks.workunit.client.0.smithi136.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2023-11-16T01:50:00.984 INFO:tasks.workunit.client.0.smithi136.stderr:Error EINVAL: invalid command 2023-11-16T01:50:00.986 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:50:00.986 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool delete foooo foooo --yes-i-really-really-mean-it 2023-11-16T01:50:01.241 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:50:01.254 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-16T01:50:01.254 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-16T01:50:01.254 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-16T01:50:01.392 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:50:01.392 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:01.392 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:01.392 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "leader", 2023-11-16T01:50:01.392 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 30, 2023-11-16T01:50:01.393 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:50:01.393 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:50:01.393 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:50:01.393 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:50:01.393 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:01.393 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 1, 2023-11-16T01:50:01.393 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:01.393 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:50:01.393 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:50:01.393 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:01.393 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:01.393 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:01.394 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:01.394 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:01.394 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:01.394 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:01.394 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:01.394 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:01.394 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:50:01.394 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:50:01.394 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:01.394 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:01.395 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:01.395 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:01.395 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:01.395 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:01.395 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:01.395 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:01.395 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.395 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.395 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:50:01.395 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:50:01.395 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:50:01.395 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:50:01.396 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:50:01.396 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:50:01.396 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:01.396 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:01.396 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:50:01.396 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:50:01.396 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:50:01.396 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:50:01.396 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:50:01.396 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:50:01.396 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:50:01.397 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:01.397 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:50:01.397 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:01.397 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:01.397 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:01.397 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:01.397 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:01.397 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:01.397 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:01.397 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:01.397 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:01.397 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:50:01.398 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.398 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:50:01.398 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.398 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:01.398 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:01.398 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:01.398 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:01.398 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.398 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:01.398 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:50:01.398 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.399 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.399 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.399 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:01.399 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:50:01.399 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.399 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.399 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.399 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.399 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:50:01.399 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:50:01.399 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:01.399 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:01.400 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:01.400 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.400 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.400 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:01.400 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:01.400 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:01.400 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:01.400 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.400 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:01.400 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:50:01.400 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.401 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.401 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.401 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:01.401 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:50:01.401 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.401 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.401 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.401 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.401 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:50:01.401 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:50:01.401 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:01.401 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:01.402 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:01.402 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.402 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.402 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:01.402 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:01.402 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:01.402 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:01.402 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.402 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:01.402 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:50:01.402 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.403 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.403 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.403 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:01.403 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:50:01.403 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.403 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.403 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.403 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.403 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:50:01.403 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:50:01.404 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:01.404 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:01.404 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:01.404 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.404 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.404 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.404 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:50:01.404 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:50:01.404 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.404 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:01.404 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:01.405 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:01.405 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.405 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:01.406 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:50:01.406 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.406 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:01.406 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:01.406 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 4 2023-11-16T01:50:01.406 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.406 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:01.407 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:50:01.407 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.407 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:01.407 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:01.407 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:01.407 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.407 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.407 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.408 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:50:01.408 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:50:01.409 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:50:01.409 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-16T01:50:01.410 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-16T01:50:01.549 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:50:01.549 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:01.549 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:01.549 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:50:01.550 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 30, 2023-11-16T01:50:01.550 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:50:01.550 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:50:01.550 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:50:01.550 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:50:01.550 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:01.550 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 1, 2023-11-16T01:50:01.550 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:01.550 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:50:01.550 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:50:01.550 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:01.550 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:01.551 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:01.551 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:01.551 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:01.551 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:01.551 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:01.551 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:01.551 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:01.551 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:50:01.551 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:50:01.551 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:01.551 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:01.552 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:01.552 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:01.552 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:01.552 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:01.552 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:01.552 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:01.552 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.552 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.552 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:50:01.552 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:50:01.553 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:50:01.553 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:50:01.553 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:50:01.553 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:50:01.553 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:01.553 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:01.558 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:50:01.561 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:50:01.561 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:50:01.562 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:50:01.562 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:50:01.562 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:50:01.562 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:50:01.562 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:01.562 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:50:01.562 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:01.562 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:01.562 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:01.562 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:01.562 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:01.563 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:01.563 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:01.563 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:01.563 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:01.563 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:50:01.563 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.563 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:50:01.574 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.574 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:01.574 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:01.574 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:01.575 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:01.575 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.575 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:01.575 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:50:01.575 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.575 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.575 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.575 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:01.575 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:50:01.575 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.575 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.575 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.576 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.576 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:50:01.576 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:50:01.576 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:01.576 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:01.576 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:01.576 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.576 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.576 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:01.576 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:01.576 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:01.576 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:01.577 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.577 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:01.577 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:50:01.577 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.577 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.577 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.577 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:01.577 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:50:01.577 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.577 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.577 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.577 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.578 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:50:01.578 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:50:01.578 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:01.578 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:01.578 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:01.578 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.578 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.578 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:01.578 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:01.578 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:01.578 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:01.578 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.579 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:01.579 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:50:01.579 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.579 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.579 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.579 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:01.579 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:50:01.579 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.579 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.579 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.579 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.580 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:50:01.580 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:50:01.580 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:01.580 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:01.580 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:01.580 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.580 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.581 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.581 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:50:01.581 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:50:01.581 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.581 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:01.581 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:01.581 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:01.581 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.581 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:01.581 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:50:01.581 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.581 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:01.582 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:01.582 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:50:01.582 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.582 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:01.582 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:50:01.582 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.582 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:01.582 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:01.582 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 3 2023-11-16T01:50:01.582 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.582 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:01.582 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": [ 2023-11-16T01:50:01.583 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.583 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:01.583 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:01.583 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:01.583 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.583 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.583 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.583 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:50:01.583 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:50:01.584 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:50:01.584 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-16T01:50:01.585 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-16T01:50:01.732 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:50:01.732 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:01.733 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:01.733 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:50:01.733 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 30, 2023-11-16T01:50:01.733 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:50:01.733 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:50:01.733 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:50:01.734 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:50:01.734 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:01.734 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 1, 2023-11-16T01:50:01.734 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:01.735 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:50:01.735 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:50:01.735 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:01.735 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:01.735 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:01.735 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:01.736 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:01.736 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:01.736 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:01.736 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:01.736 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:01.736 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:50:01.736 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:50:01.736 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:01.737 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:01.737 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:01.737 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:01.737 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:01.737 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:01.737 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:01.737 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:01.738 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.738 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.738 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:50:01.739 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:50:01.739 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:50:01.739 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:50:01.739 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:50:01.740 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:50:01.740 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:01.740 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:01.740 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:50:01.741 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:50:01.741 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:50:01.741 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:50:01.741 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:50:01.742 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:50:01.742 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:50:01.742 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:01.742 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:50:01.742 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:01.743 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:01.743 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:01.743 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:01.743 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:01.743 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:01.743 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:01.744 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:01.744 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:01.744 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:50:01.744 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.744 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:50:01.744 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.745 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:01.745 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:01.745 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:01.745 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:01.745 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.745 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:01.745 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:50:01.746 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.746 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.746 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.746 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:01.746 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:50:01.747 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.747 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.747 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.747 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.747 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:50:01.748 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:50:01.748 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:01.748 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:01.748 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:01.748 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.748 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.749 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:01.749 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:01.749 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:01.749 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:01.749 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.749 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:01.749 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:50:01.750 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.750 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.750 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.750 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:01.750 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:50:01.750 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.750 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.751 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.751 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.751 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:50:01.751 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:50:01.751 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:01.751 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:01.752 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:01.752 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.752 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.753 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:01.753 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:01.753 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:01.753 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:01.753 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.753 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:01.754 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:50:01.754 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.754 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.754 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.754 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:01.754 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:50:01.755 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:01.755 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.755 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.755 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.755 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:50:01.756 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:50:01.756 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:01.756 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:01.756 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:01.756 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.757 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.757 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.757 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:50:01.757 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:50:01.757 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.758 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:01.758 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:01.758 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:01.758 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.758 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:01.759 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:50:01.759 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:01.759 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:01.759 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:01.759 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:50:01.759 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:01.760 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:01.760 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:01.760 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:50:01.760 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:50:01.762 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:50:01.762 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-11-16T01:50:01.762 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-11-16T01:50:02.124 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.120+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:50:02.124 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.120+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=12,kv=6,logm=68} crc {auth=4084353969,config=3140514009,health=4074779856,kv=2868953028,logm=1027320809}) 2023-11-16T01:50:02.124 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.120+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=12,kv=6,logm=67} crc {auth=802491100,config=3140514009,health=2789173196,kv=2868953028,logm=393393460}) 2023-11-16T01:50:02.124 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.120+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:50:02.124 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.120+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=12,kv=6,logm=68} crc {auth=4084353969,config=3140514009,health=4074779856,kv=2868953028,logm=1027320809}) 2023-11-16T01:50:02.125 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.120+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=11,kv=6,logm=68} crc {auth=802491100,config=3140514009,health=1372817,kv=2868953028,logm=2435146518}) 2023-11-16T01:50:02.134 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.128+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:50:02.134 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.128+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=3,monmap=3,osd_metadata=4} crc {logm=3007874165,mdsmap=1531151339,mgr=3338689344,mgr_command_descs=1448462394,mgr_metadata=2889854584,mgrstat=2477503980,mon_config_key=1707289356,monmap=1549841071,osd_metadata=1781502970}) 2023-11-16T01:50:02.134 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.128+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=67,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=3,monmap=3,osd_metadata=4} crc {logm=1117278412,mdsmap=1531151339,mgr=3338689344,mgr_command_descs=1448462394,mgr_metadata=2889854584,mgrstat=3939799532,mon_config_key=1707289356,monmap=1549841071,osd_metadata=1781502970}) 2023-11-16T01:50:02.134 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.128+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:50:02.134 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.128+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=3,monmap=3,osd_metadata=4} crc {logm=3007874165,mdsmap=1531151339,mgr=3338689344,mgr_command_descs=1448462394,mgr_metadata=2889854584,mgrstat=2477503980,mon_config_key=1707289356,monmap=1549841071,osd_metadata=1781502970}) 2023-11-16T01:50:02.134 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.128+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=67,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=3,monmap=3,osd_metadata=5} crc {logm=111531878,mdsmap=3585761405,mgr=3338689344,mgr_command_descs=1448462394,mgr_metadata=2889854584,mgrstat=3939799532,mon_config_key=1707289356,monmap=1549841071,osd_metadata=3582743870}) 2023-11-16T01:50:02.138 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.136+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:50:02.138 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.136+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osdmap=79} crc {osd_metadata=812545599,osd_pg_creating=5029610,osdmap=312666552}) 2023-11-16T01:50:02.138 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.136+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osdmap=80} crc {osd_metadata=812545599,osd_pg_creating=5029610,osdmap=1920863915}) 2023-11-16T01:50:02.138 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.136+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:50:02.138 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.136+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osdmap=79} crc {osd_metadata=812545599,osd_pg_creating=5029610,osdmap=312666552}) 2023-11-16T01:50:02.138 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:02.136+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osdmap=82} crc {osd_metadata=812545599,osd_pg_creating=5029610,osdmap=4045380471}) 2023-11-16T01:50:02.139 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-11-16T01:50:02.159 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'foooo' does not exist 2023-11-16T01:50:02.167 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2023-11-16T01:50:03.140 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-16T01:50:03.140 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:50:03.166 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'fooo' does not exist 2023-11-16T01:50:03.177 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2023-11-16T01:50:03.475 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:50:03.488 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-16T01:50:03.488 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-11-16T01:50:03.488 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-16T01:50:03.489 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-16T01:50:03.621 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:50:03.621 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:03.621 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:03.621 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "leader", 2023-11-16T01:50:03.622 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 30, 2023-11-16T01:50:03.622 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:50:03.622 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:50:03.622 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:50:03.622 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:50:03.622 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:03.622 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 3, 2023-11-16T01:50:03.622 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:03.622 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:50:03.622 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:50:03.622 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:03.622 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:03.623 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:03.623 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:03.623 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:03.623 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:03.623 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:03.623 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:03.623 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:03.623 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:50:03.623 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:50:03.623 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:03.623 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:03.624 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:03.624 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:03.624 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:03.624 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:03.624 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:03.624 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:03.624 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.624 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.624 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:50:03.624 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:50:03.624 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:50:03.625 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:50:03.625 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:50:03.625 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:50:03.625 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:03.625 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:03.625 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:50:03.625 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:50:03.625 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:50:03.626 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:50:03.626 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:50:03.626 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:50:03.626 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:50:03.626 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:03.626 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:50:03.626 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:03.627 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:03.627 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:03.627 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:03.627 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:03.627 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:03.627 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:03.627 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:03.627 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:03.627 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:50:03.627 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.627 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:50:03.627 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.628 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:03.628 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:03.628 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:03.628 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:03.628 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.628 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:03.628 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:50:03.628 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.628 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.628 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.629 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:03.629 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:50:03.629 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.629 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.629 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.629 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.629 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:50:03.629 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:50:03.629 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:03.629 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:03.629 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:03.629 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.630 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.630 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:03.630 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:03.630 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:03.630 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:03.630 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.630 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:03.630 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:50:03.630 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.631 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.631 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.631 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:03.631 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:50:03.631 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.631 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.631 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.631 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.631 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:50:03.631 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:50:03.631 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:03.631 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:03.632 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:03.632 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.632 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.632 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:03.632 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:03.632 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:03.632 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:03.632 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.632 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:03.632 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:50:03.632 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.632 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.633 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.633 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:03.633 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:50:03.633 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.633 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.633 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.633 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.633 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:50:03.633 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:50:03.633 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:03.634 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:03.634 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:03.634 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.634 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.634 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.634 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:50:03.634 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:50:03.634 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.634 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:03.634 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:03.634 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:03.634 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.635 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:03.635 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:50:03.635 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.635 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:03.635 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:03.635 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 4 2023-11-16T01:50:03.635 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.635 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:03.635 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:50:03.635 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.635 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:03.635 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:03.636 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:03.636 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.636 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.636 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.636 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:50:03.636 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:50:03.640 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:50:03.641 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-16T01:50:03.641 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-16T01:50:03.788 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:50:03.788 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:03.789 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:03.789 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:50:03.789 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 30, 2023-11-16T01:50:03.789 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:50:03.789 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:50:03.789 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:50:03.789 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:50:03.789 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:03.789 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 3, 2023-11-16T01:50:03.789 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:03.789 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:50:03.790 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:50:03.790 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:03.790 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:03.790 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:03.790 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:03.790 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:03.790 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:03.790 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:03.790 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:03.790 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:03.790 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:50:03.790 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:50:03.791 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:03.791 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:03.791 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:03.791 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:03.791 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:03.791 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:03.791 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:03.791 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:03.791 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.791 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.791 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:50:03.792 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:50:03.792 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:50:03.792 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:50:03.792 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:50:03.792 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:50:03.792 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:03.792 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:03.792 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:50:03.792 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:50:03.792 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:50:03.792 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:50:03.792 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:50:03.793 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:50:03.793 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:50:03.794 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:03.794 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:50:03.794 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:03.794 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:03.794 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:03.794 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:03.794 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:03.794 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:03.794 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:03.794 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:03.795 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:03.795 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:50:03.795 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.795 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:50:03.795 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.795 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:03.795 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:03.795 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:03.795 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:03.795 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.795 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:03.796 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:50:03.796 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.796 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.796 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.796 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:03.796 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:50:03.796 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.797 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.797 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.797 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.797 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:50:03.797 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:50:03.797 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:03.797 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:03.797 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:03.797 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.797 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.797 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:03.797 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:03.798 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:03.798 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:03.798 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.798 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:03.798 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:50:03.798 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.798 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.798 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.798 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:03.798 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:50:03.798 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.799 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.799 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.799 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.799 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:50:03.799 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:50:03.799 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:03.799 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:03.799 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:03.799 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.799 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.799 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:03.800 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:03.800 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:03.800 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:03.800 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.800 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:03.800 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:50:03.800 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.800 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.800 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.800 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:03.800 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:50:03.800 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.801 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.801 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.801 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.801 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:50:03.801 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:50:03.801 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:03.801 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:03.801 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:03.801 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.801 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.801 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.801 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:50:03.802 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:50:03.802 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.802 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:03.802 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:03.802 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:03.802 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.802 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:03.802 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:50:03.802 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.802 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:03.802 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:03.802 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:50:03.803 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.803 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:03.803 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:50:03.803 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.803 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:03.803 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:03.803 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 3 2023-11-16T01:50:03.803 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.803 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:03.803 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": [ 2023-11-16T01:50:03.803 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.803 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:03.804 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:03.804 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:03.804 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.804 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.804 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.804 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:50:03.804 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:50:03.807 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:50:03.807 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-16T01:50:03.807 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-16T01:50:03.952 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:50:03.952 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:03.952 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:03.952 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:50:03.952 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 30, 2023-11-16T01:50:03.953 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:50:03.953 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:50:03.953 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:50:03.953 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:50:03.953 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:03.953 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 4, 2023-11-16T01:50:03.954 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:03.954 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:50:03.954 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:50:03.954 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:03.954 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:03.954 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:03.955 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:03.955 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:03.955 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:03.955 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:03.955 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:03.955 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:03.955 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:50:03.956 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:50:03.956 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:03.956 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:03.956 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:03.956 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:03.956 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:03.956 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:03.956 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:03.957 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:03.957 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.957 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.957 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:50:03.957 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:50:03.957 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:50:03.957 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:50:03.958 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:50:03.958 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:50:03.958 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:03.958 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:03.958 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:50:03.958 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:50:03.958 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:50:03.958 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:50:03.959 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:50:03.959 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:50:03.959 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:50:03.959 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:03.959 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:50:03.959 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:03.960 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:03.961 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:03.962 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:03.962 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:03.962 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:03.962 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:03.962 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:03.962 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:03.962 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:50:03.962 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.962 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:50:03.963 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.963 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:03.963 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:03.963 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:03.963 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:03.963 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.963 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:03.963 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:50:03.963 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.963 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.964 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.964 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:03.964 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:50:03.964 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.964 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.964 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.965 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.965 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:50:03.965 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:50:03.965 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:03.965 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:03.965 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:03.965 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.965 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.965 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:03.965 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:03.966 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:03.966 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:03.966 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.966 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:03.966 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:50:03.966 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.967 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.967 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.967 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:03.967 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:50:03.967 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.967 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.970 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.970 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.971 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:50:03.971 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:50:03.971 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:03.971 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:03.971 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:03.971 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.971 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.971 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:03.971 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:03.972 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:03.972 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:03.972 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.972 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:03.972 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:50:03.972 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.972 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.973 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.973 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:03.973 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:50:03.973 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:03.973 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.973 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.973 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.973 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:50:03.973 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:50:03.973 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:03.974 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:03.974 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:03.974 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.974 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.974 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.974 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:50:03.975 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:50:03.975 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.975 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:03.975 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:03.975 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:03.975 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.975 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:03.976 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:50:03.976 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:03.976 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:03.976 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:03.976 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:50:03.976 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:03.977 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:03.977 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:03.977 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:50:03.977 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:50:03.978 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:50:03.978 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-11-16T01:50:03.979 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-11-16T01:50:03.979 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-11-16T01:50:03.980 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-11-16T01:50:03.980 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-16T01:50:03.980 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T01:50:03.998 INFO:tasks.ceph.mon.b:Stopped 2023-11-16T01:50:03.999 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-11-16T01:50:03.999 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:50:04.331 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:50:07.346 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:50:07.687 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:50:10.703 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:50:11.056 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:50:14.072 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:50:18.650 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:50:18.664 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-11-16T01:50:18.664 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-11-16T01:50:18.664 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-16T01:50:18.799 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:50:18.800 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:18.800 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:18.800 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "leader", 2023-11-16T01:50:18.800 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 34, 2023-11-16T01:50:18.800 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:50:18.800 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:50:18.801 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:50:18.801 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:18.801 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 0, 2023-11-16T01:50:18.801 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:18.801 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:50:18.802 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:50:18.802 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:18.802 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:18.802 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:18.802 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:18.802 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:18.803 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:18.803 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:18.803 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:18.803 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:18.803 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:50:18.804 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:50:18.804 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:18.804 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:18.804 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:18.804 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:18.804 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:18.805 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:18.805 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:18.805 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:18.805 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:18.805 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:18.806 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:50:18.806 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:50:18.806 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:50:18.806 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:50:18.806 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:50:18.807 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:50:18.807 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:18.807 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:18.807 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:50:18.807 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:50:18.807 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:50:18.808 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:50:18.808 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:50:18.808 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:50:18.808 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:50:18.808 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:18.809 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:50:18.809 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:18.809 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:18.809 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:18.809 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:18.809 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:18.810 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:18.810 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:18.810 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:18.810 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:18.810 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:50:18.810 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:18.811 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:50:18.811 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:18.811 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:18.811 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:18.811 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:18.811 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:18.812 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:18.812 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:18.814 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:50:18.814 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:18.814 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:18.814 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:18.815 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:18.815 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:50:18.815 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:18.815 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:18.815 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:18.816 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:18.816 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:50:18.816 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:50:18.816 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:18.816 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:18.816 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:18.816 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:18.817 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:18.817 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:18.817 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:18.817 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:18.817 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:18.817 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:18.818 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:18.818 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:50:18.818 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:18.818 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:18.818 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:18.818 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:18.818 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:50:18.819 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:18.819 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:18.819 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:18.819 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:18.819 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:50:18.819 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:50:18.820 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:18.820 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:18.820 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:18.820 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:18.820 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:18.820 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:18.820 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:18.821 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:18.821 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:18.821 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:18.821 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:18.821 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:50:18.821 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:18.821 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:18.821 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:18.822 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:18.822 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:50:18.822 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:18.822 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:18.822 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:18.822 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:18.823 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:50:18.823 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:50:18.823 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:18.823 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:18.823 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:18.823 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:18.823 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:18.824 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:18.824 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:50:18.824 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:50:18.824 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:18.825 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:18.825 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:18.825 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:18.825 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:18.825 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:18.825 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:50:18.825 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:18.826 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:18.826 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:18.826 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 4 2023-11-16T01:50:18.826 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:18.826 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:18.826 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:50:18.826 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:18.827 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:18.827 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:18.827 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:18.827 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:18.836 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:18.836 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:18.836 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:50:18.836 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:50:18.840 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-16T01:50:18.841 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-11-16T01:50:18.841 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-16T01:50:18.989 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:50:18.989 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:18.990 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:18.990 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:50:18.990 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 34, 2023-11-16T01:50:18.990 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:50:18.990 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:50:18.990 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:50:18.990 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:18.990 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 0, 2023-11-16T01:50:18.990 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:18.990 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:50:18.990 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:50:18.991 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:18.991 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:18.991 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:18.991 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:18.991 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:18.991 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:18.991 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:18.991 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:18.991 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:18.991 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:50:18.991 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:50:18.991 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:18.992 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:18.992 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:18.992 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:18.992 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:18.992 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:18.992 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:18.992 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:18.992 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:18.993 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:19.000 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:50:19.000 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:50:19.000 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:50:19.001 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:50:19.001 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:50:19.001 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:50:19.001 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:19.001 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:19.001 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:50:19.002 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:50:19.002 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:50:19.002 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:50:19.002 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:50:19.002 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:50:19.002 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:50:19.003 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:19.003 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:50:19.003 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:19.003 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:19.003 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:19.003 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:19.004 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:19.004 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:19.004 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:19.004 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:19.004 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:19.004 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:50:19.005 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:19.005 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:50:19.005 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:19.005 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:19.005 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:19.006 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:19.006 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:19.006 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:19.006 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:19.006 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:50:19.007 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:19.007 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:19.007 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:19.007 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:19.007 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:50:19.008 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:19.008 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:19.008 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:19.008 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:19.008 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:50:19.008 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:50:19.009 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:19.009 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:19.009 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:19.009 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:19.009 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:19.009 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:19.009 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:19.010 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:19.010 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:19.010 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:19.010 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:19.010 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:50:19.010 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:19.015 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:19.015 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:19.015 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:19.016 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:50:19.016 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:19.016 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:19.016 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:19.017 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:19.017 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:50:19.017 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:50:19.017 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:19.017 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:19.018 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:19.018 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:19.018 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:19.018 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:19.018 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:19.019 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:19.019 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:19.019 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:19.019 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:19.019 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:50:19.019 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:19.020 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:19.020 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:19.020 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:19.020 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:50:19.020 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:19.020 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:19.020 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:19.021 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:19.021 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:50:19.021 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:50:19.021 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:19.021 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:19.021 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:19.021 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:19.022 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:19.022 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:19.022 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:50:19.022 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:50:19.022 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:19.023 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:19.023 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:19.023 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:19.023 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:19.023 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:19.023 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:50:19.024 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:19.024 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:19.024 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:19.024 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:50:19.024 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:19.024 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:19.025 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:50:19.025 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:19.025 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:19.025 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:19.025 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 5 2023-11-16T01:50:19.026 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:19.027 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:19.027 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": [ 2023-11-16T01:50:19.027 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:19.027 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:19.028 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:19.028 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:19.028 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:19.028 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:19.029 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:19.029 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:50:19.029 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:50:19.031 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-16T01:50:19.031 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-11-16T01:50:19.530 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'foo' does not exist 2023-11-16T01:50:19.538 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2023-11-16T01:50:19.914 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'foo' does not exist 2023-11-16T01:50:19.921 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2023-11-16T01:50:21.133 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'fooo' does not exist 2023-11-16T01:50:21.141 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2023-11-16T01:50:21.523 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'fooo' does not exist 2023-11-16T01:50:21.531 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool delete fuggg fuggg --yes-i-really-really-mean-it 2023-11-16T01:50:21.892 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'fuggg' does not exist 2023-11-16T01:50:21.899 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo OK 2023-11-16T01:50:21.900 INFO:tasks.workunit.client.0.smithi136.stdout:OK 2023-11-16T01:50:21.901 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T01:50:21.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T01:50:21.911 INFO:tasks.workunit:Running workunits matching mon/crush_ops.sh on client.0... 2023-11-16T01:50:21.912 INFO:tasks.workunit:Running workunit mon/crush_ops.sh... 2023-11-16T01:50:21.912 DEBUG:teuthology.orchestra.run.smithi136:workunit test mon/crush_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ed3c8e926b9de93026c8a7c1d86be5d6821b52ac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh 2023-11-16T01:50:21.924 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush dump 2023-11-16T01:50:22.292 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2023-11-16T01:50:22.292 INFO:tasks.workunit.client.0.smithi136.stdout: "devices": [ 2023-11-16T01:50:22.292 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.293 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 0, 2023-11-16T01:50:22.293 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "osd.0", 2023-11-16T01:50:22.293 INFO:tasks.workunit.client.0.smithi136.stdout: "class": "ssd" 2023-11-16T01:50:22.293 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.293 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.293 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 1, 2023-11-16T01:50:22.294 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "osd.1", 2023-11-16T01:50:22.294 INFO:tasks.workunit.client.0.smithi136.stdout: "class": "ssd" 2023-11-16T01:50:22.294 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.294 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.294 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 2, 2023-11-16T01:50:22.295 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "osd.2", 2023-11-16T01:50:22.295 INFO:tasks.workunit.client.0.smithi136.stdout: "class": "ssd" 2023-11-16T01:50:22.295 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.295 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.295 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 3, 2023-11-16T01:50:22.295 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "osd.3", 2023-11-16T01:50:22.296 INFO:tasks.workunit.client.0.smithi136.stdout: "class": "ssd" 2023-11-16T01:50:22.296 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.296 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.296 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 4, 2023-11-16T01:50:22.296 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "osd.4", 2023-11-16T01:50:22.296 INFO:tasks.workunit.client.0.smithi136.stdout: "class": "ssd" 2023-11-16T01:50:22.297 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.297 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.297 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 5, 2023-11-16T01:50:22.297 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "osd.5", 2023-11-16T01:50:22.297 INFO:tasks.workunit.client.0.smithi136.stdout: "class": "ssd" 2023-11-16T01:50:22.297 INFO:tasks.workunit.client.0.smithi136.stdout: } 2023-11-16T01:50:22.297 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2023-11-16T01:50:22.297 INFO:tasks.workunit.client.0.smithi136.stdout: "types": [ 2023-11-16T01:50:22.298 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.299 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 0, 2023-11-16T01:50:22.299 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "osd" 2023-11-16T01:50:22.299 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.299 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.299 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 1, 2023-11-16T01:50:22.300 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "host" 2023-11-16T01:50:22.300 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.300 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.300 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 2, 2023-11-16T01:50:22.300 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "chassis" 2023-11-16T01:50:22.300 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.301 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.301 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 3, 2023-11-16T01:50:22.301 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "rack" 2023-11-16T01:50:22.301 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.301 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.301 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 4, 2023-11-16T01:50:22.302 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "row" 2023-11-16T01:50:22.302 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.302 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.302 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 5, 2023-11-16T01:50:22.302 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "pdu" 2023-11-16T01:50:22.302 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.303 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.303 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 6, 2023-11-16T01:50:22.303 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "pod" 2023-11-16T01:50:22.303 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.303 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.303 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 7, 2023-11-16T01:50:22.304 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "room" 2023-11-16T01:50:22.304 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.304 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.304 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 8, 2023-11-16T01:50:22.304 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "datacenter" 2023-11-16T01:50:22.304 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.304 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.304 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 9, 2023-11-16T01:50:22.305 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "zone" 2023-11-16T01:50:22.305 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.305 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.305 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 10, 2023-11-16T01:50:22.305 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "region" 2023-11-16T01:50:22.305 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.306 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.306 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 11, 2023-11-16T01:50:22.306 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "root" 2023-11-16T01:50:22.306 INFO:tasks.workunit.client.0.smithi136.stdout: } 2023-11-16T01:50:22.306 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2023-11-16T01:50:22.306 INFO:tasks.workunit.client.0.smithi136.stdout: "buckets": [ 2023-11-16T01:50:22.307 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.307 INFO:tasks.workunit.client.0.smithi136.stdout: "id": -1, 2023-11-16T01:50:22.307 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "default", 2023-11-16T01:50:22.307 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 11, 2023-11-16T01:50:22.307 INFO:tasks.workunit.client.0.smithi136.stdout: "type_name": "root", 2023-11-16T01:50:22.307 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 34560, 2023-11-16T01:50:22.308 INFO:tasks.workunit.client.0.smithi136.stdout: "alg": "straw2", 2023-11-16T01:50:22.308 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": "rjenkins1", 2023-11-16T01:50:22.308 INFO:tasks.workunit.client.0.smithi136.stdout: "items": [ 2023-11-16T01:50:22.308 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.308 INFO:tasks.workunit.client.0.smithi136.stdout: "id": -3, 2023-11-16T01:50:22.308 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 17280, 2023-11-16T01:50:22.308 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 0 2023-11-16T01:50:22.308 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.308 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.309 INFO:tasks.workunit.client.0.smithi136.stdout: "id": -5, 2023-11-16T01:50:22.309 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 17280, 2023-11-16T01:50:22.309 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 1 2023-11-16T01:50:22.309 INFO:tasks.workunit.client.0.smithi136.stdout: } 2023-11-16T01:50:22.309 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2023-11-16T01:50:22.309 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.309 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.309 INFO:tasks.workunit.client.0.smithi136.stdout: "id": -2, 2023-11-16T01:50:22.309 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "default~ssd", 2023-11-16T01:50:22.309 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 11, 2023-11-16T01:50:22.309 INFO:tasks.workunit.client.0.smithi136.stdout: "type_name": "root", 2023-11-16T01:50:22.309 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 34560, 2023-11-16T01:50:22.310 INFO:tasks.workunit.client.0.smithi136.stdout: "alg": "straw2", 2023-11-16T01:50:22.310 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": "rjenkins1", 2023-11-16T01:50:22.310 INFO:tasks.workunit.client.0.smithi136.stdout: "items": [ 2023-11-16T01:50:22.310 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.310 INFO:tasks.workunit.client.0.smithi136.stdout: "id": -4, 2023-11-16T01:50:22.310 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 17280, 2023-11-16T01:50:22.310 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 0 2023-11-16T01:50:22.311 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.311 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.311 INFO:tasks.workunit.client.0.smithi136.stdout: "id": -6, 2023-11-16T01:50:22.311 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 17280, 2023-11-16T01:50:22.311 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 1 2023-11-16T01:50:22.311 INFO:tasks.workunit.client.0.smithi136.stdout: } 2023-11-16T01:50:22.312 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2023-11-16T01:50:22.312 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.312 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.312 INFO:tasks.workunit.client.0.smithi136.stdout: "id": -3, 2023-11-16T01:50:22.312 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "smithi083", 2023-11-16T01:50:22.312 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 1, 2023-11-16T01:50:22.312 INFO:tasks.workunit.client.0.smithi136.stdout: "type_name": "host", 2023-11-16T01:50:22.312 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 17280, 2023-11-16T01:50:22.312 INFO:tasks.workunit.client.0.smithi136.stdout: "alg": "straw2", 2023-11-16T01:50:22.313 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": "rjenkins1", 2023-11-16T01:50:22.313 INFO:tasks.workunit.client.0.smithi136.stdout: "items": [ 2023-11-16T01:50:22.313 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.313 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 0, 2023-11-16T01:50:22.313 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 5760, 2023-11-16T01:50:22.313 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 0 2023-11-16T01:50:22.313 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.313 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.313 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 1, 2023-11-16T01:50:22.313 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 5760, 2023-11-16T01:50:22.313 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 1 2023-11-16T01:50:22.314 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.314 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.314 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 2, 2023-11-16T01:50:22.314 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 5760, 2023-11-16T01:50:22.314 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 2 2023-11-16T01:50:22.314 INFO:tasks.workunit.client.0.smithi136.stdout: } 2023-11-16T01:50:22.314 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2023-11-16T01:50:22.314 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.314 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.315 INFO:tasks.workunit.client.0.smithi136.stdout: "id": -4, 2023-11-16T01:50:22.315 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "smithi083~ssd", 2023-11-16T01:50:22.315 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 1, 2023-11-16T01:50:22.315 INFO:tasks.workunit.client.0.smithi136.stdout: "type_name": "host", 2023-11-16T01:50:22.315 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 17280, 2023-11-16T01:50:22.315 INFO:tasks.workunit.client.0.smithi136.stdout: "alg": "straw2", 2023-11-16T01:50:22.315 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": "rjenkins1", 2023-11-16T01:50:22.315 INFO:tasks.workunit.client.0.smithi136.stdout: "items": [ 2023-11-16T01:50:22.316 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.316 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 0, 2023-11-16T01:50:22.316 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 5760, 2023-11-16T01:50:22.316 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 0 2023-11-16T01:50:22.316 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.316 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.316 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 1, 2023-11-16T01:50:22.316 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 5760, 2023-11-16T01:50:22.316 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 1 2023-11-16T01:50:22.316 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.316 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.317 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 2, 2023-11-16T01:50:22.317 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 5760, 2023-11-16T01:50:22.317 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 2 2023-11-16T01:50:22.317 INFO:tasks.workunit.client.0.smithi136.stdout: } 2023-11-16T01:50:22.317 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2023-11-16T01:50:22.317 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.317 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.317 INFO:tasks.workunit.client.0.smithi136.stdout: "id": -5, 2023-11-16T01:50:22.317 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "smithi136", 2023-11-16T01:50:22.317 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 1, 2023-11-16T01:50:22.318 INFO:tasks.workunit.client.0.smithi136.stdout: "type_name": "host", 2023-11-16T01:50:22.318 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 17280, 2023-11-16T01:50:22.318 INFO:tasks.workunit.client.0.smithi136.stdout: "alg": "straw2", 2023-11-16T01:50:22.318 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": "rjenkins1", 2023-11-16T01:50:22.318 INFO:tasks.workunit.client.0.smithi136.stdout: "items": [ 2023-11-16T01:50:22.318 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.318 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 3, 2023-11-16T01:50:22.318 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 5760, 2023-11-16T01:50:22.318 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 0 2023-11-16T01:50:22.319 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.319 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.319 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 5, 2023-11-16T01:50:22.319 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 5760, 2023-11-16T01:50:22.319 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 1 2023-11-16T01:50:22.319 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.319 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.319 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 4, 2023-11-16T01:50:22.319 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 5760, 2023-11-16T01:50:22.319 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 2 2023-11-16T01:50:22.319 INFO:tasks.workunit.client.0.smithi136.stdout: } 2023-11-16T01:50:22.320 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2023-11-16T01:50:22.320 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.320 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.320 INFO:tasks.workunit.client.0.smithi136.stdout: "id": -6, 2023-11-16T01:50:22.320 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "smithi136~ssd", 2023-11-16T01:50:22.320 INFO:tasks.workunit.client.0.smithi136.stdout: "type_id": 1, 2023-11-16T01:50:22.320 INFO:tasks.workunit.client.0.smithi136.stdout: "type_name": "host", 2023-11-16T01:50:22.320 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 17280, 2023-11-16T01:50:22.320 INFO:tasks.workunit.client.0.smithi136.stdout: "alg": "straw2", 2023-11-16T01:50:22.320 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": "rjenkins1", 2023-11-16T01:50:22.321 INFO:tasks.workunit.client.0.smithi136.stdout: "items": [ 2023-11-16T01:50:22.321 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.321 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 3, 2023-11-16T01:50:22.321 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 5760, 2023-11-16T01:50:22.321 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 0 2023-11-16T01:50:22.321 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.321 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.321 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 5, 2023-11-16T01:50:22.321 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 5760, 2023-11-16T01:50:22.322 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 1 2023-11-16T01:50:22.322 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.322 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.322 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 4, 2023-11-16T01:50:22.322 INFO:tasks.workunit.client.0.smithi136.stdout: "weight": 5760, 2023-11-16T01:50:22.323 INFO:tasks.workunit.client.0.smithi136.stdout: "pos": 2 2023-11-16T01:50:22.323 INFO:tasks.workunit.client.0.smithi136.stdout: } 2023-11-16T01:50:22.323 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2023-11-16T01:50:22.323 INFO:tasks.workunit.client.0.smithi136.stdout: } 2023-11-16T01:50:22.323 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2023-11-16T01:50:22.324 INFO:tasks.workunit.client.0.smithi136.stdout: "rules": [ 2023-11-16T01:50:22.324 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.324 INFO:tasks.workunit.client.0.smithi136.stdout: "rule_id": 0, 2023-11-16T01:50:22.324 INFO:tasks.workunit.client.0.smithi136.stdout: "rule_name": "replicated_rule", 2023-11-16T01:50:22.324 INFO:tasks.workunit.client.0.smithi136.stdout: "ruleset": 0, 2023-11-16T01:50:22.324 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 1, 2023-11-16T01:50:22.324 INFO:tasks.workunit.client.0.smithi136.stdout: "min_size": 1, 2023-11-16T01:50:22.324 INFO:tasks.workunit.client.0.smithi136.stdout: "max_size": 10, 2023-11-16T01:50:22.324 INFO:tasks.workunit.client.0.smithi136.stdout: "steps": [ 2023-11-16T01:50:22.324 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.324 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "take", 2023-11-16T01:50:22.324 INFO:tasks.workunit.client.0.smithi136.stdout: "item": -1, 2023-11-16T01:50:22.325 INFO:tasks.workunit.client.0.smithi136.stdout: "item_name": "default" 2023-11-16T01:50:22.325 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.325 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.326 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "choose_firstn", 2023-11-16T01:50:22.326 INFO:tasks.workunit.client.0.smithi136.stdout: "num": 0, 2023-11-16T01:50:22.326 INFO:tasks.workunit.client.0.smithi136.stdout: "type": "osd" 2023-11-16T01:50:22.326 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.326 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.326 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "emit" 2023-11-16T01:50:22.327 INFO:tasks.workunit.client.0.smithi136.stdout: } 2023-11-16T01:50:22.327 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2023-11-16T01:50:22.327 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.327 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.327 INFO:tasks.workunit.client.0.smithi136.stdout: "rule_id": 1, 2023-11-16T01:50:22.327 INFO:tasks.workunit.client.0.smithi136.stdout: "rule_name": "erasure-code", 2023-11-16T01:50:22.328 INFO:tasks.workunit.client.0.smithi136.stdout: "ruleset": 1, 2023-11-16T01:50:22.328 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 3, 2023-11-16T01:50:22.328 INFO:tasks.workunit.client.0.smithi136.stdout: "min_size": 3, 2023-11-16T01:50:22.328 INFO:tasks.workunit.client.0.smithi136.stdout: "max_size": 3, 2023-11-16T01:50:22.328 INFO:tasks.workunit.client.0.smithi136.stdout: "steps": [ 2023-11-16T01:50:22.328 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.328 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "set_chooseleaf_tries", 2023-11-16T01:50:22.329 INFO:tasks.workunit.client.0.smithi136.stdout: "num": 5 2023-11-16T01:50:22.329 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.329 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.329 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "set_choose_tries", 2023-11-16T01:50:22.329 INFO:tasks.workunit.client.0.smithi136.stdout: "num": 100 2023-11-16T01:50:22.330 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.330 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.330 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "take", 2023-11-16T01:50:22.330 INFO:tasks.workunit.client.0.smithi136.stdout: "item": -1, 2023-11-16T01:50:22.330 INFO:tasks.workunit.client.0.smithi136.stdout: "item_name": "default" 2023-11-16T01:50:22.330 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.330 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.330 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "choose_indep", 2023-11-16T01:50:22.331 INFO:tasks.workunit.client.0.smithi136.stdout: "num": 0, 2023-11-16T01:50:22.331 INFO:tasks.workunit.client.0.smithi136.stdout: "type": "osd" 2023-11-16T01:50:22.331 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.331 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.331 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "emit" 2023-11-16T01:50:22.331 INFO:tasks.workunit.client.0.smithi136.stdout: } 2023-11-16T01:50:22.331 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2023-11-16T01:50:22.332 INFO:tasks.workunit.client.0.smithi136.stdout: } 2023-11-16T01:50:22.332 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2023-11-16T01:50:22.332 INFO:tasks.workunit.client.0.smithi136.stdout: "tunables": { 2023-11-16T01:50:22.332 INFO:tasks.workunit.client.0.smithi136.stdout: "choose_local_tries": 0, 2023-11-16T01:50:22.332 INFO:tasks.workunit.client.0.smithi136.stdout: "choose_local_fallback_tries": 0, 2023-11-16T01:50:22.332 INFO:tasks.workunit.client.0.smithi136.stdout: "choose_total_tries": 50, 2023-11-16T01:50:22.333 INFO:tasks.workunit.client.0.smithi136.stdout: "chooseleaf_descend_once": 1, 2023-11-16T01:50:22.333 INFO:tasks.workunit.client.0.smithi136.stdout: "chooseleaf_vary_r": 1, 2023-11-16T01:50:22.333 INFO:tasks.workunit.client.0.smithi136.stdout: "chooseleaf_stable": 1, 2023-11-16T01:50:22.333 INFO:tasks.workunit.client.0.smithi136.stdout: "straw_calc_version": 1, 2023-11-16T01:50:22.333 INFO:tasks.workunit.client.0.smithi136.stdout: "allowed_bucket_algs": 54, 2023-11-16T01:50:22.333 INFO:tasks.workunit.client.0.smithi136.stdout: "profile": "jewel", 2023-11-16T01:50:22.334 INFO:tasks.workunit.client.0.smithi136.stdout: "optimal_tunables": 1, 2023-11-16T01:50:22.334 INFO:tasks.workunit.client.0.smithi136.stdout: "legacy_tunables": 0, 2023-11-16T01:50:22.334 INFO:tasks.workunit.client.0.smithi136.stdout: "minimum_required_version": "jewel", 2023-11-16T01:50:22.334 INFO:tasks.workunit.client.0.smithi136.stdout: "require_feature_tunables": 1, 2023-11-16T01:50:22.334 INFO:tasks.workunit.client.0.smithi136.stdout: "require_feature_tunables2": 1, 2023-11-16T01:50:22.334 INFO:tasks.workunit.client.0.smithi136.stdout: "has_v2_rules": 1, 2023-11-16T01:50:22.335 INFO:tasks.workunit.client.0.smithi136.stdout: "require_feature_tunables3": 1, 2023-11-16T01:50:22.335 INFO:tasks.workunit.client.0.smithi136.stdout: "has_v3_rules": 0, 2023-11-16T01:50:22.335 INFO:tasks.workunit.client.0.smithi136.stdout: "has_v4_buckets": 1, 2023-11-16T01:50:22.335 INFO:tasks.workunit.client.0.smithi136.stdout: "require_feature_tunables5": 1, 2023-11-16T01:50:22.335 INFO:tasks.workunit.client.0.smithi136.stdout: "has_v5_rules": 0 2023-11-16T01:50:22.335 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.335 INFO:tasks.workunit.client.0.smithi136.stdout: "choose_args": {} 2023-11-16T01:50:22.335 INFO:tasks.workunit.client.0.smithi136.stdout:} 2023-11-16T01:50:22.335 INFO:tasks.workunit.client.0.smithi136.stdout: 2023-11-16T01:50:22.336 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule dump 2023-11-16T01:50:22.669 INFO:tasks.workunit.client.0.smithi136.stdout:[ 2023-11-16T01:50:22.669 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.669 INFO:tasks.workunit.client.0.smithi136.stdout: "rule_id": 0, 2023-11-16T01:50:22.669 INFO:tasks.workunit.client.0.smithi136.stdout: "rule_name": "replicated_rule", 2023-11-16T01:50:22.669 INFO:tasks.workunit.client.0.smithi136.stdout: "ruleset": 0, 2023-11-16T01:50:22.669 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 1, 2023-11-16T01:50:22.669 INFO:tasks.workunit.client.0.smithi136.stdout: "min_size": 1, 2023-11-16T01:50:22.669 INFO:tasks.workunit.client.0.smithi136.stdout: "max_size": 10, 2023-11-16T01:50:22.669 INFO:tasks.workunit.client.0.smithi136.stdout: "steps": [ 2023-11-16T01:50:22.670 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.670 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "take", 2023-11-16T01:50:22.670 INFO:tasks.workunit.client.0.smithi136.stdout: "item": -1, 2023-11-16T01:50:22.670 INFO:tasks.workunit.client.0.smithi136.stdout: "item_name": "default" 2023-11-16T01:50:22.670 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.670 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.670 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "choose_firstn", 2023-11-16T01:50:22.670 INFO:tasks.workunit.client.0.smithi136.stdout: "num": 0, 2023-11-16T01:50:22.670 INFO:tasks.workunit.client.0.smithi136.stdout: "type": "osd" 2023-11-16T01:50:22.670 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.670 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.671 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "emit" 2023-11-16T01:50:22.671 INFO:tasks.workunit.client.0.smithi136.stdout: } 2023-11-16T01:50:22.671 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2023-11-16T01:50:22.671 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.671 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.671 INFO:tasks.workunit.client.0.smithi136.stdout: "rule_id": 1, 2023-11-16T01:50:22.671 INFO:tasks.workunit.client.0.smithi136.stdout: "rule_name": "erasure-code", 2023-11-16T01:50:22.671 INFO:tasks.workunit.client.0.smithi136.stdout: "ruleset": 1, 2023-11-16T01:50:22.671 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 3, 2023-11-16T01:50:22.671 INFO:tasks.workunit.client.0.smithi136.stdout: "min_size": 3, 2023-11-16T01:50:22.671 INFO:tasks.workunit.client.0.smithi136.stdout: "max_size": 3, 2023-11-16T01:50:22.671 INFO:tasks.workunit.client.0.smithi136.stdout: "steps": [ 2023-11-16T01:50:22.672 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.672 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "set_chooseleaf_tries", 2023-11-16T01:50:22.672 INFO:tasks.workunit.client.0.smithi136.stdout: "num": 5 2023-11-16T01:50:22.672 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.672 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.672 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "set_choose_tries", 2023-11-16T01:50:22.672 INFO:tasks.workunit.client.0.smithi136.stdout: "num": 100 2023-11-16T01:50:22.672 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.672 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.672 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "take", 2023-11-16T01:50:22.672 INFO:tasks.workunit.client.0.smithi136.stdout: "item": -1, 2023-11-16T01:50:22.672 INFO:tasks.workunit.client.0.smithi136.stdout: "item_name": "default" 2023-11-16T01:50:22.672 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.673 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.673 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "choose_indep", 2023-11-16T01:50:22.673 INFO:tasks.workunit.client.0.smithi136.stdout: "num": 0, 2023-11-16T01:50:22.673 INFO:tasks.workunit.client.0.smithi136.stdout: "type": "osd" 2023-11-16T01:50:22.673 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2023-11-16T01:50:22.673 INFO:tasks.workunit.client.0.smithi136.stdout: { 2023-11-16T01:50:22.673 INFO:tasks.workunit.client.0.smithi136.stdout: "op": "emit" 2023-11-16T01:50:22.673 INFO:tasks.workunit.client.0.smithi136.stdout: } 2023-11-16T01:50:22.673 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2023-11-16T01:50:22.673 INFO:tasks.workunit.client.0.smithi136.stdout: } 2023-11-16T01:50:22.673 INFO:tasks.workunit.client.0.smithi136.stdout:] 2023-11-16T01:50:22.673 INFO:tasks.workunit.client.0.smithi136.stdout: 2023-11-16T01:50:22.680 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule ls 2023-11-16T01:50:23.870 INFO:tasks.workunit.client.0.smithi136.stdout:replicated_rule 2023-11-16T01:50:26.832 INFO:tasks.workunit.client.0.smithi136.stdout:erasure-code 2023-11-16T01:50:26.833 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule list 2023-11-16T01:50:26.839 INFO:tasks.workunit.client.0.smithi136.stdout:replicated_rule 2023-11-16T01:50:26.840 INFO:tasks.workunit.client.0.smithi136.stdout:erasure-code 2023-11-16T01:50:26.840 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule create-simple foo default host 2023-11-16T01:50:26.840 INFO:tasks.workunit.client.0.smithi136.stderr:rule foo already exists 2023-11-16T01:50:26.840 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule create-simple foo default host 2023-11-16T01:50:26.840 INFO:tasks.workunit.client.0.smithi136.stderr:2023-11-16T01:50:25.880+0000 7f3adb7fe700 0 --2- 172.21.15.136:0/2573316093 >> v2:172.21.15.83:3300/0 conn(0x7f3adc09e390 0x7f3adc1336e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-11-16T01:50:26.978 INFO:tasks.workunit.client.0.smithi136.stderr:rule foo already exists 2023-11-16T01:50:26.986 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule create-simple bar default host 2023-11-16T01:50:27.570 INFO:tasks.workunit.client.0.smithi136.stderr:rule bar already exists 2023-11-16T01:50:27.581 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rm-device-class all 2023-11-16T01:50:29.609 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2023-11-16T01:50:29.623 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush set-device-class ssd osd.0 2023-11-16T01:50:31.627 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2023-11-16T01:50:31.640 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush set-device-class hdd osd.1 2023-11-16T01:50:33.388 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2023-11-16T01:50:33.401 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule create-replicated foo-ssd default host ssd 2023-11-16T01:50:34.776 INFO:tasks.workunit.client.0.smithi136.stderr:rule foo-ssd already exists 2023-11-16T01:50:34.783 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule create-replicated foo-hdd default host hdd 2023-11-16T01:50:35.782 INFO:tasks.workunit.client.0.smithi136.stderr:rule foo-hdd already exists 2023-11-16T01:50:35.791 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule ls-by-class ssd 2023-11-16T01:50:35.792 INFO:tasks.workunit.client.0.smithi136.stderr:+ grep foo-ssd 2023-11-16T01:50:36.178 INFO:tasks.workunit.client.0.smithi136.stdout:foo-ssd 2023-11-16T01:50:36.179 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule ls-by-class ssd 2023-11-16T01:50:36.179 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false grep foo-hdd 2023-11-16T01:50:36.179 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:50:36.179 INFO:tasks.workunit.client.0.smithi136.stderr:+ grep foo-hdd 2023-11-16T01:50:36.568 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:50:36.569 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule ls-by-class hdd 2023-11-16T01:50:36.569 INFO:tasks.workunit.client.0.smithi136.stderr:+ grep foo-hdd 2023-11-16T01:50:36.933 INFO:tasks.workunit.client.0.smithi136.stdout:foo-hdd 2023-11-16T01:50:36.934 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule ls-by-class hdd 2023-11-16T01:50:36.934 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect_false grep foo-ssd 2023-11-16T01:50:36.934 INFO:tasks.workunit.client.0.smithi136.stderr:+ set -x 2023-11-16T01:50:36.934 INFO:tasks.workunit.client.0.smithi136.stderr:+ grep foo-ssd 2023-11-16T01:50:37.315 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 0 2023-11-16T01:50:37.315 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd erasure-code-profile set ec-foo-ssd crush-device-class=ssd m=2 k=2 2023-11-16T01:50:38.002 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool create ec-foo 2 erasure ec-foo-ssd 2023-11-16T01:50:39.032 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-11-16T01:50:39.032 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-11-16T01:50:39.032 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-16T01:50:39.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-16T01:50:39.036 INFO:tasks.ceph.mon.b:Started 2023-11-16T01:50:39.036 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-16T01:50:39.036 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:50:39.374 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:50:42.394 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:50:49.486 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ec-foo' already exists 2023-11-16T01:50:49.500 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool rm ec-foo ec-foo --yes-i-really-really-mean-it 2023-11-16T01:50:49.715 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:50:49.729 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-16T01:50:49.729 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-16T01:50:49.730 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-16T01:50:49.811 INFO:tasks.workunit.client.0.smithi136.stderr:2023-11-16T01:50:49.808+0000 7fc9b1d5a700 0 --2- 172.21.15.136:0/3915873444 >> v2:172.21.15.136:3300/0 conn(0x7fc9ac090ba0 0x7fc9ac12ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-11-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "leader", 2023-11-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 46, 2023-11-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 0, 2023-11-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:49.874 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:50:49.874 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 4 2023-11-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:50:49.896 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:50:49.896 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-16T01:50:49.896 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-16T01:50:50.034 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:50:50.034 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:50.034 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:50.034 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:50:50.034 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 46, 2023-11-16T01:50:50.035 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:50:50.035 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:50:50.035 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:50:50.035 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:50:50.035 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:50.035 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 0, 2023-11-16T01:50:50.035 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:50.035 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:50:50.035 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:50:50.035 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:50.035 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:50.036 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:50.036 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:50.036 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:50.036 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:50.036 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:50.036 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:50.036 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:50.036 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:50:50.036 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:50:50.036 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:50.036 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:50.036 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:50.037 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:50.037 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:50.037 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:50.037 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:50.037 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:50.037 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:50.037 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.037 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:50:50.037 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:50:50.037 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:50:50.037 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:50:50.037 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:50:50.037 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:50:50.038 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:50.038 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:50.039 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:50:50.039 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:50:50.039 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:50:50.039 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:50:50.039 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:50:50.039 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:50:50.039 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:50:50.039 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:50.039 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:50:50.040 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:50.040 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:50.040 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:50.040 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:50.040 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:50.040 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:50.040 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:50.040 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:50.040 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:50.040 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:50:50.040 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.041 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:50:50.041 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.041 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:50.041 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:50.041 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:50.041 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:50.041 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.041 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:50.041 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:50:50.041 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:50.041 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.041 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.042 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:50.042 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:50:50.042 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:50.042 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:50.042 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:50.042 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.042 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:50:50.042 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:50:50.042 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:50.042 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:50.042 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:50.042 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.043 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.043 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:50.043 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:50.043 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:50.043 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:50.043 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.043 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:50.043 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:50:50.043 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:50.043 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.043 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.043 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:50.043 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:50:50.044 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:50.044 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:50.044 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:50.044 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.044 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:50:50.044 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:50:50.044 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:50.044 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:50.044 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:50.045 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.045 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.045 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:50.045 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:50.045 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:50.045 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:50.045 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.045 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:50.045 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:50:50.045 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:50.045 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.045 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.046 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:50.046 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:50:50.046 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:50.046 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:50.046 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:50.046 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.046 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:50:50.046 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:50:50.046 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:50.046 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:50.046 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:50.047 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:50.047 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:50.047 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.047 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:50:50.047 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:50:50.047 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.047 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:50.047 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:50.047 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:50.047 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:50.047 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:50.047 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:50:50.048 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.048 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:50.048 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:50.048 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:50:50.048 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:50.048 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:50.048 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:50:50.048 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.048 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:50.048 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:50.048 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 3 2023-11-16T01:50:50.049 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:50.049 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:50.049 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": [ 2023-11-16T01:50:50.049 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.049 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:50.049 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:50.049 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:50.050 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:50.050 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:50.050 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.050 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:50:50.050 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:50:50.051 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:50:50.051 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-16T01:50:50.051 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-16T01:50:50.194 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:50:50.194 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:50.194 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:50.194 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:50:50.194 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 46, 2023-11-16T01:50:50.194 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:50:50.195 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:50:50.195 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:50:50.195 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:50:50.195 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:50.195 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 0, 2023-11-16T01:50:50.195 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:50.195 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:50:50.195 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:50:50.195 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:50.195 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:50.195 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:50.195 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:50.196 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:50.196 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:50.196 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:50.196 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:50.196 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:50.196 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:50:50.196 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:50:50.196 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:50.196 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:50.196 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:50.196 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:50.196 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:50.196 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:50.197 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:50.197 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:50.197 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:50.197 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.197 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:50:50.197 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:50:50.197 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:50:50.197 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:50:50.198 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:50:50.198 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:50:50.198 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:50.198 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:50.198 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:50:50.198 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:50:50.198 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:50:50.198 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:50:50.198 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:50:50.198 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:50:50.199 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:50:50.199 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:50.199 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:50:50.199 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:50.199 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:50.199 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:50.199 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:50.199 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:50.199 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:50.199 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:50.199 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:50.199 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:50.200 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:50:50.200 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.200 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:50:50.200 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.200 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:50.200 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:50.200 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:50.200 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:50.200 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.200 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:50.200 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:50:50.200 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:50.200 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.201 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.201 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:50.201 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:50:50.201 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:50.201 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:50.201 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:50.201 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.202 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:50:50.202 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:50:50.202 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:50.202 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:50.202 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:50.202 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.202 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.202 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:50.202 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:50.202 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:50.202 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:50.202 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.202 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:50.203 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:50:50.203 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:50.203 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.203 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.203 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:50.203 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:50:50.203 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:50.203 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:50.203 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:50.203 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.203 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:50:50.203 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:50:50.203 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:50.204 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:50.204 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:50.204 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.204 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.204 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:50.211 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:50.211 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:50.211 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:50:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:50:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:50:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:50:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:50:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:50:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:50:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:50:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:50:50.219 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:50:50.220 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:50:50.220 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-11-16T01:50:50.220 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-11-16T01:50:50.584 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:50.579+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:50:50.584 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:50.579+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=6,logm=67} crc {auth=3463054422,config=3140514009,health=2471807558,kv=2868953028,logm=3694250796}) 2023-11-16T01:50:50.585 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:50.579+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=5,logm=67} crc {auth=462866510,config=3140514009,health=2425708035,kv=3700160801,logm=3041387437}) 2023-11-16T01:50:50.585 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:50.579+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:50:50.585 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:50.579+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=6,logm=67} crc {auth=3463054422,config=3140514009,health=2471807558,kv=2868953028,logm=3694250796}) 2023-11-16T01:50:50.585 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:50.579+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=12,kv=6,logm=66} crc {auth=462866510,config=3140514009,health=2425708035,kv=2868953028,logm=492938923}) 2023-11-16T01:50:50.597 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-11-16T01:50:50.599 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:50.595+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:50:50.599 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:50.595+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1171673722}) 2023-11-16T01:50:50.599 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:50.595+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2788487163}) 2023-11-16T01:50:50.599 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:50.595+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:50:50.599 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:50.595+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1171673722}) 2023-11-16T01:50:50.599 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:50.595+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=592607018}) 2023-11-16T01:50:50.604 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ec-foo' does not exist 2023-11-16T01:50:50.616 INFO:tasks.workunit.client.0.smithi136.stderr:+ grep foo 2023-11-16T01:50:50.616 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule ls 2023-11-16T01:50:50.996 INFO:tasks.workunit.client.0.smithi136.stdout:foo 2023-11-16T01:50:50.997 INFO:tasks.workunit.client.0.smithi136.stdout:foo-ssd 2023-11-16T01:50:50.997 INFO:tasks.workunit.client.0.smithi136.stdout:foo-hdd 2023-11-16T01:50:50.997 INFO:tasks.workunit.client.0.smithi136.stdout:ec-foo 2023-11-16T01:50:50.997 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule rename foo foo-asdf 2023-11-16T01:50:51.333 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:51.331+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:50:51.333 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:51.331+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=9,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=54} crc {logm=1913706712,mdsmap=1531151339,mgr=3338689344,mgr_command_descs=1448462394,mgr_metadata=2889854584,mgrstat=3209533606,mon_config_key=1707289356,monmap=1549841071,osd_metadata=2408539844,osd_pg_creating=496959941,osdmap=725022873}) 2023-11-16T01:50:51.333 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:51.331+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=9,mdsmap=3,mgr=6,mgr_metadata=1,mgrstat=12,mon_config_key=3,monmap=3,osd_metadata=6,osdmap=57} crc {logm=1913706712,mdsmap=1531151339,mgr=992179106,mgr_metadata=2889854584,mgrstat=3209533606,mon_config_key=1707289356,monmap=1549841071,osd_metadata=2408539844,osdmap=98470813}) 2023-11-16T01:50:51.333 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:51.331+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:50:51.334 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:51.331+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=9,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=54} crc {logm=1913706712,mdsmap=1531151339,mgr=3338689344,mgr_command_descs=1448462394,mgr_metadata=2889854584,mgrstat=3209533606,mon_config_key=1707289356,monmap=1549841071,osd_metadata=2408539844,osd_pg_creating=496959941,osdmap=725022873}) 2023-11-16T01:50:51.334 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:51.331+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=9,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=3,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=55} crc {logm=1913706712,mdsmap=1531151339,mgr=3338689344,mgr_command_descs=1448462394,mgr_metadata=2889854584,mgrstat=3209533606,mon_config_key=1707289356,monmap=1549841071,osd_metadata=2605294420,osd_pg_creating=496959941,osdmap=3179341644}) 2023-11-16T01:50:51.346 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:51.339+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:50:51.346 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:51.343+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=2055782978}) 2023-11-16T01:50:51.346 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:51.343+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=51} crc {osdmap=4280247863}) 2023-11-16T01:50:51.346 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:51.343+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:50:51.347 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:51.343+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=2055782978}) 2023-11-16T01:50:51.347 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:50:51.343+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=49} crc {osdmap=1348665935}) 2023-11-16T01:50:51.598 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-16T01:50:51.598 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:50:51.706 INFO:tasks.workunit.client.0.smithi136.stderr:already renamed to 'foo-asdf' 2023-11-16T01:50:51.714 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule rename foo foo-asdf 2023-11-16T01:50:51.933 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:50:51.947 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-16T01:50:51.947 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-11-16T01:50:51.947 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-16T01:50:51.947 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-16T01:50:52.073 INFO:tasks.workunit.client.0.smithi136.stderr:already renamed to 'foo-asdf' 2023-11-16T01:50:52.081 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule rename bar bar-asdf 2023-11-16T01:50:52.086 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:50:52.086 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:52.086 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:52.086 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "leader", 2023-11-16T01:50:52.087 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 46, 2023-11-16T01:50:52.087 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:50:52.087 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:50:52.087 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:50:52.087 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:50:52.087 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:52.087 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 2, 2023-11-16T01:50:52.087 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:52.087 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:50:52.088 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:50:52.088 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:52.088 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:52.088 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:52.088 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:52.088 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:52.088 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:52.088 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:52.088 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:52.088 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:52.088 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:50:52.089 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:50:52.089 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:52.089 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:52.089 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:52.089 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:52.089 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:52.089 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:52.089 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:52.089 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:52.089 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.090 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.090 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:50:52.090 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:50:52.090 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:50:52.090 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:50:52.090 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:50:52.090 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:50:52.090 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:52.090 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:52.090 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:50:52.091 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:50:52.091 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:50:52.091 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:50:52.091 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:50:52.093 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:50:52.093 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:50:52.093 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:52.093 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:50:52.093 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:52.093 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:52.093 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:52.093 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:52.094 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:52.094 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:52.094 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:52.094 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:52.094 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:52.094 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:50:52.094 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.094 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:50:52.095 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.095 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:52.095 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:52.095 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:52.095 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:52.095 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.095 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:52.095 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:50:52.095 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.095 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.095 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.096 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:52.096 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:50:52.096 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.096 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.096 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.096 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.096 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:50:52.096 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:50:52.096 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:52.096 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:52.096 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:52.096 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.097 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.097 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:52.097 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:52.097 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:52.098 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:52.098 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.098 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:52.098 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:50:52.099 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.099 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.099 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.099 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:52.099 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:50:52.103 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.103 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.103 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.104 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.104 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:50:52.104 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:50:52.105 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:52.105 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:52.105 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:52.105 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.105 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.105 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:52.105 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:52.105 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:52.105 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:52.105 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:50:52.107 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:50:52.107 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:52.107 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:52.107 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:52.107 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.107 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.107 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.107 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:50:52.107 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:50:52.107 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.107 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:52.107 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:52.107 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:52.108 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.108 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:52.108 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:50:52.108 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.108 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:52.108 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:52.108 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 4 2023-11-16T01:50:52.108 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.108 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:52.108 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:50:52.109 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.109 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:52.109 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:52.109 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:50:52.109 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.109 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.109 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.109 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:50:52.109 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:50:52.110 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:50:52.110 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-16T01:50:52.111 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-16T01:50:52.260 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 46, 2023-11-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 2, 2023-11-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 4 2023-11-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": [ 2023-11-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:50:52.278 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:50:52.278 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-16T01:50:52.278 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-16T01:50:52.428 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:50:52.428 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:52.428 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:52.428 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:50:52.428 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 46, 2023-11-16T01:50:52.429 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:50:52.429 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:50:52.429 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:50:52.429 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:50:52.429 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:52.429 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 2, 2023-11-16T01:50:52.429 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:52.429 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:50:52.429 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:50:52.429 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:52.429 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:52.429 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:52.430 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:52.430 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:52.430 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:52.430 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:52.430 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:52.430 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:52.430 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:50:52.430 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:50:52.430 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:52.430 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:52.430 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:52.430 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:52.431 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:52.431 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:52.431 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:52.431 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:52.431 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.431 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.431 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:50:52.431 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:50:52.431 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:50:52.431 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:50:52.431 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:50:52.431 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:50:52.432 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:52.432 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:50:52.432 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:50:52.432 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:50:52.432 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:50:52.432 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:50:52.432 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:50:52.433 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:50:52.433 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:50:52.433 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:50:52.433 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:50:52.433 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:50:52.433 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:50:52.433 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:50:52.433 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:50:52.433 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:50:52.433 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:50:52.433 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:50:52.433 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:50:52.434 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:52.434 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:50:52.434 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.434 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:50:52.434 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.434 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:50:52.434 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:50:52.434 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:52.434 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:52.434 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.434 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:52.434 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:50:52.435 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.435 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.435 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.435 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:52.435 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:50:52.435 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.435 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.435 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.435 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.435 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:50:52.435 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:50:52.435 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:52.436 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:52.436 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:52.436 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.436 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.436 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:50:52.436 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:50:52.436 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:52.436 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:52.436 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.436 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:52.436 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:50:52.436 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.437 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.437 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.437 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:52.437 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:50:52.437 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.437 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.437 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.437 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.437 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:50:52.437 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:50:52.437 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:52.437 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:52.438 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:52.438 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.438 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.438 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:50:52.438 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:50:52.438 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:50:52.438 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:50:52.438 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.438 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:50:52.438 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:50:52.438 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.438 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.439 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.439 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:50:52.439 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:50:52.439 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:50:52.439 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.439 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.439 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.439 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:50:52.439 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:50:52.439 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:50:52.439 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:50:52.439 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:50:52.440 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.440 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.440 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.440 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:50:52.440 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:50:52.440 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.440 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:52.440 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:52.440 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:50:52.440 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.440 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:50:52.440 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:50:52.440 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:50:52.441 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:50:52.441 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:50:52.441 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:50:52.441 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:50:52.441 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:50:52.441 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:50:52.441 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:50:52.441 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:50:52.443 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:50:52.443 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-11-16T01:50:52.443 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2023-11-16T01:50:52.443 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-11-16T01:50:52.443 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-11-16T01:50:52.443 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-16T01:50:52.443 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T01:50:52.460 INFO:tasks.ceph.mon.b:Stopped 2023-11-16T01:50:52.460 INFO:tasks.mon_thrash:Sending STOP to mon c 2023-11-16T01:50:52.460 INFO:tasks.ceph.mon.c:Sent signal 19 2023-11-16T01:50:52.460 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-11-16T01:50:52.558 INFO:tasks.workunit.client.0.smithi136.stderr:already renamed to 'bar-asdf' 2023-11-16T01:50:52.567 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule ls 2023-11-16T01:50:52.567 INFO:tasks.workunit.client.0.smithi136.stderr:+ grep foo-asdf 2023-11-16T01:50:52.951 INFO:tasks.workunit.client.0.smithi136.stdout:foo-asdf 2023-11-16T01:50:52.951 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd crush rule ls 2023-11-16T01:50:52.952 INFO:tasks.workunit.client.0.smithi136.stderr:+ grep bar-asdf 2023-11-16T01:50:53.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:50:53.341 INFO:tasks.workunit:Stopping ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh'] on client.0... 2023-11-16T01:50:53.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-11-16T01:50:53.588 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/workunit.py", line 423, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test mon/crush_ops.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ed3c8e926b9de93026c8a7c1d86be5d6821b52ac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh' 2023-11-16T01:50:53.674 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=89688c83dd364ce2aa0daaf0410e9434 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/workunit.py", line 423, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test mon/crush_ops.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ed3c8e926b9de93026c8a7c1d86be5d6821b52ac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh' 2023-11-16T01:50:53.676 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2023-11-16T01:50:53.686 INFO:tasks.mon_thrash:joining mon_thrasher 2023-11-16T01:51:07.461 INFO:tasks.mon_thrash:Sending CONT to mon c 2023-11-16T01:51:07.462 INFO:tasks.ceph.mon.c:Sent signal 18 2023-11-16T01:51:07.462 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-11-16T01:51:07.463 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:51:12.755 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":48,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:51:12.769 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-11-16T01:51:12.769 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-11-16T01:51:12.769 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-16T01:51:12.905 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:51:12.905 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:51:12.905 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:51:12.905 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "leader", 2023-11-16T01:51:12.905 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 48, 2023-11-16T01:51:12.906 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:51:12.906 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:51:12.906 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:51:12.906 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:12.906 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 0, 2023-11-16T01:51:12.906 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:51:12.906 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:51:12.906 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:51:12.907 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:51:12.907 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:51:12.907 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:51:12.907 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:51:12.907 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:51:12.907 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:51:12.907 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:51:12.908 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:51:12.908 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:12.908 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:51:12.908 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:51:12.908 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:51:12.908 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:51:12.908 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:51:12.908 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:51:12.909 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:51:12.909 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:51:12.909 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:51:12.909 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:51:12.909 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:12.909 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:12.909 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:51:12.909 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:51:12.910 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:51:12.910 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:51:12.910 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:51:12.910 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:51:12.910 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:51:12.910 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:51:12.910 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:51:12.911 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:51:12.911 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:51:12.911 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:51:12.911 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:51:12.911 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:51:12.911 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:51:12.911 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:51:12.911 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:51:12.912 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:51:12.912 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:51:12.912 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:51:12.912 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:51:12.912 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:51:12.912 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:51:12.912 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:51:12.912 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:51:12.913 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:12.913 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:51:12.913 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:12.913 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:51:12.913 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:12.913 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:51:12.913 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:51:12.914 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:51:12.914 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:51:12.914 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:12.914 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:51:12.914 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:51:12.914 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:12.914 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:12.914 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:12.915 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:51:12.915 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:51:12.915 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:12.915 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:12.915 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:12.915 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:12.915 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:51:12.916 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:51:12.916 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:51:12.916 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:51:12.916 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:51:12.916 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:12.916 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:12.916 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:51:12.916 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:51:12.917 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:51:12.917 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:51:12.917 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:12.917 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:51:12.917 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:51:12.917 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:12.917 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:12.917 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:12.918 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:51:12.918 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:51:12.918 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:12.918 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:12.918 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:12.918 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:12.918 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:51:12.919 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:51:12.919 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:51:12.919 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:51:12.919 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:51:12.919 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:12.919 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:12.919 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:51:12.919 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:51:12.920 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:51:12.920 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:51:12.920 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:12.920 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:51:12.920 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:51:12.920 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:12.920 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:12.921 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:12.921 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:51:12.921 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:51:12.921 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:12.921 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:12.921 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:12.921 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:12.921 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:51:12.922 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:51:12.922 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:51:12.922 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:51:12.922 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:51:12.922 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:12.923 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:12.923 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:12.923 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:51:12.923 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:51:12.923 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:12.924 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:12.924 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:12.925 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:51:12.925 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:12.925 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:12.925 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:51:12.925 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:12.925 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:12.925 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:12.926 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 4 2023-11-16T01:51:12.926 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:12.926 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:12.926 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:51:12.926 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:12.926 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:12.926 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:12.927 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:51:12.927 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:12.927 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:12.927 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:12.927 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:51:12.927 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:51:12.928 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-16T01:51:12.928 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-11-16T01:51:12.929 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-16T01:51:13.070 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:51:13.071 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:51:13.071 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:51:13.071 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:51:13.071 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 48, 2023-11-16T01:51:13.072 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:51:13.072 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:51:13.072 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:51:13.072 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:13.073 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 0, 2023-11-16T01:51:13.073 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:51:13.073 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:51:13.073 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:51:13.073 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:51:13.074 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:51:13.074 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:51:13.074 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:51:13.074 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:51:13.075 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:51:13.075 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:51:13.075 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:51:13.075 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:13.075 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:51:13.075 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:51:13.075 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:51:13.075 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:51:13.076 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:51:13.076 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:51:13.076 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:51:13.076 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:51:13.076 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:51:13.076 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:51:13.076 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:13.077 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:13.077 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:51:13.077 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:51:13.077 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:51:13.077 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:51:13.077 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:51:13.077 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:51:13.078 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:51:13.078 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:51:13.078 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:51:13.078 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:51:13.078 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:51:13.078 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:51:13.078 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:51:13.078 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:51:13.079 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:51:13.079 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:51:13.079 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:51:13.079 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:51:13.079 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:51:13.079 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:51:13.079 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:51:13.080 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:51:13.080 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:51:13.080 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:51:13.080 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:51:13.080 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:13.080 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:51:13.080 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:13.080 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:51:13.081 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:13.081 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:51:13.081 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:51:13.081 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:51:13.081 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:51:13.081 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:13.081 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:51:13.081 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:51:13.082 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:13.082 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:13.082 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:13.082 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:51:13.082 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:51:13.082 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:13.082 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:13.082 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:13.083 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:13.083 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:51:13.083 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:51:13.083 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:51:13.083 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:51:13.083 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:51:13.083 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:13.084 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:13.084 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:51:13.084 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:51:13.084 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:51:13.084 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:51:13.084 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:13.084 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:51:13.084 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:51:13.085 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:13.085 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:13.085 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:13.085 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:51:13.085 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:51:13.085 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:13.085 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:13.086 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:13.086 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:13.086 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:51:13.086 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:51:13.086 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:51:13.086 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:51:13.086 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:51:13.086 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:13.087 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:13.087 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:51:13.087 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:51:13.087 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:51:13.087 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:51:13.087 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:13.087 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:51:13.088 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:51:13.088 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:13.088 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:13.088 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:13.088 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:51:13.088 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:51:13.088 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:13.088 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:13.089 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:13.089 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:13.089 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:51:13.089 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:51:13.089 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:51:13.089 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:51:13.089 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:51:13.089 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:13.090 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:13.090 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:13.090 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:51:13.090 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:51:13.090 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:13.090 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:13.090 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:13.091 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:51:13.091 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:13.091 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:13.091 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:51:13.091 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:13.091 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:13.091 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:13.092 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:51:13.092 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:13.092 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:13.092 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:51:13.092 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:13.092 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:13.092 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:13.092 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 4 2023-11-16T01:51:13.093 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:13.093 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:13.093 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": [ 2023-11-16T01:51:13.093 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:13.093 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:13.093 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:13.093 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:51:13.094 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:13.094 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:13.094 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:13.094 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:51:13.094 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:51:13.095 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-16T01:51:13.095 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-11-16T01:51:33.097 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-11-16T01:51:33.097 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-11-16T01:51:33.098 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-16T01:51:33.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-16T01:51:33.101 INFO:tasks.ceph.mon.b:Started 2023-11-16T01:51:33.101 INFO:tasks.mon_thrash:Sending STOP to mon c 2023-11-16T01:51:33.101 INFO:tasks.ceph.mon.c:Sent signal 19 2023-11-16T01:51:33.101 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-11-16T01:51:48.102 INFO:tasks.mon_thrash:Sending CONT to mon c 2023-11-16T01:51:48.103 INFO:tasks.ceph.mon.c:Sent signal 18 2023-11-16T01:51:48.103 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-16T01:51:48.103 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:51:48.447 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:51:48.460 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-16T01:51:48.461 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-16T01:51:48.461 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-16T01:51:48.599 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:51:48.599 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:51:48.599 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:51:48.599 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "leader", 2023-11-16T01:51:48.599 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 52, 2023-11-16T01:51:48.600 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:51:48.600 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:51:48.600 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:51:48.600 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:51:48.600 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:48.600 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 0, 2023-11-16T01:51:48.601 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:51:48.601 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:51:48.601 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:51:48.601 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:51:48.601 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:51:48.601 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:51:48.602 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:51:48.602 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:51:48.602 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:51:48.602 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:51:48.602 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:51:48.602 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:48.603 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:51:48.603 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:51:48.603 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:51:48.603 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:51:48.603 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:51:48.604 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:51:48.604 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:51:48.604 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:51:48.604 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:51:48.604 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:51:48.605 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:48.605 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.605 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:51:48.605 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:51:48.605 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:51:48.605 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:51:48.606 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:51:48.606 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:51:48.606 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:51:48.606 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:51:48.606 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:51:48.606 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:51:48.607 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:51:48.607 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:51:48.608 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:51:48.608 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:51:48.608 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:51:48.608 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:51:48.608 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:51:48.608 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:51:48.609 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:51:48.609 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:51:48.609 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:51:48.609 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:51:48.609 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:51:48.610 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:51:48.610 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:51:48.610 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:48.610 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:51:48.611 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.611 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:51:48.611 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.611 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:51:48.611 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:51:48.612 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:51:48.612 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:51:48.612 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.612 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:51:48.612 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:51:48.613 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:48.613 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.613 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.613 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:51:48.613 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:51:48.613 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:48.614 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:48.614 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:48.614 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.614 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:51:48.614 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:51:48.615 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:51:48.615 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:51:48.615 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:51:48.615 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.615 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.615 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:51:48.616 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:51:48.616 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:51:48.616 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:51:48.616 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.617 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:51:48.617 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:51:48.617 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:48.617 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.617 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.618 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:51:48.618 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:51:48.618 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:48.618 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:48.619 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:48.619 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.619 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:51:48.619 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:51:48.619 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:51:48.619 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:51:48.620 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:51:48.620 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.620 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.620 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:51:48.621 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:51:48.621 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:51:48.621 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:51:48.621 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.621 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:51:48.621 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:51:48.622 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:48.622 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.622 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.622 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:51:48.622 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:51:48.623 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:48.623 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:48.623 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:48.623 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.623 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:51:48.623 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:51:48.624 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:51:48.624 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:51:48.624 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:51:48.624 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:48.624 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:48.624 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.625 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:51:48.625 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:51:48.625 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.625 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:48.625 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:48.625 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:51:48.626 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:48.626 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:48.626 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:51:48.626 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.626 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:48.627 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:48.627 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 4 2023-11-16T01:51:48.627 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:48.627 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:48.627 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:51:48.627 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.628 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:48.628 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:48.629 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:51:48.629 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:48.629 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:48.629 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.629 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:51:48.630 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:51:48.631 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:51:48.631 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-16T01:51:48.631 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-16T01:51:48.773 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:51:48.773 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:51:48.774 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:51:48.774 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:51:48.774 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 52, 2023-11-16T01:51:48.774 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:51:48.774 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:51:48.775 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:51:48.775 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:51:48.775 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:48.775 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 0, 2023-11-16T01:51:48.775 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:51:48.775 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:51:48.775 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:51:48.775 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:51:48.776 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:51:48.776 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:51:48.776 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:51:48.776 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:51:48.776 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:51:48.776 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:51:48.776 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:51:48.776 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:48.777 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:51:48.777 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:51:48.777 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:51:48.777 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:51:48.777 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:51:48.777 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:51:48.777 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:51:48.777 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:51:48.778 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:51:48.778 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:51:48.778 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:48.778 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.778 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:51:48.778 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:51:48.778 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:51:48.778 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:51:48.779 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:51:48.779 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:51:48.779 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:51:48.779 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:51:48.779 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:51:48.779 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:51:48.779 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:51:48.779 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:51:48.779 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:51:48.780 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:51:48.780 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:51:48.780 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:51:48.780 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:51:48.780 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:51:48.780 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:51:48.780 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:51:48.780 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:51:48.781 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:51:48.781 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:51:48.781 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:51:48.781 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:51:48.781 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:48.781 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:51:48.781 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.781 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:51:48.782 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.782 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:51:48.782 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:51:48.782 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:51:48.783 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:51:48.783 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.783 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:51:48.783 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:51:48.783 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:48.783 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.783 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.783 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:51:48.784 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:51:48.784 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:48.784 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:48.784 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:48.784 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.784 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:51:48.784 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:51:48.785 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:51:48.785 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:51:48.785 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:51:48.785 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.785 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.785 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:51:48.785 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:51:48.785 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:51:48.785 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:51:48.786 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.786 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:51:48.786 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:51:48.786 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:48.786 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.786 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.786 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:51:48.786 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:51:48.787 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:48.787 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:48.787 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:48.787 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.787 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:51:48.787 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:51:48.787 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:51:48.787 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:51:48.788 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:51:48.788 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.788 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.788 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:51:48.788 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:51:48.788 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:51:48.788 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:51:48.788 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.789 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:51:48.789 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:51:48.789 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:48.789 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.789 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.789 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:51:48.789 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:51:48.789 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:48.790 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:48.790 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:48.792 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.792 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:51:48.792 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:51:48.793 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:51:48.793 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:51:48.793 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:51:48.793 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:48.793 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:48.793 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.794 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:51:48.794 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:51:48.794 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.794 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:48.794 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:48.794 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:51:48.794 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:48.795 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:48.795 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": [ 2023-11-16T01:51:48.795 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.795 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:48.795 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:48.795 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 2 2023-11-16T01:51:48.795 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:48.796 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:48.796 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:51:48.796 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.796 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:48.796 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:48.796 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 3 2023-11-16T01:51:48.796 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:48.796 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:48.797 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": [ 2023-11-16T01:51:48.797 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:48.797 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:48.797 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:48.797 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:51:48.797 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:48.798 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:48.798 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:48.798 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:51:48.798 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:51:48.799 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:51:48.799 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-16T01:51:48.800 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-16T01:51:49.149 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:51:49.150 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:51:49.150 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:51:49.150 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "peon", 2023-11-16T01:51:49.150 INFO:teuthology.orchestra.run.smithi083.stdout: "election_epoch": 52, 2023-11-16T01:51:49.150 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum": [ 2023-11-16T01:51:49.150 INFO:teuthology.orchestra.run.smithi083.stdout: 0, 2023-11-16T01:51:49.151 INFO:teuthology.orchestra.run.smithi083.stdout: 1, 2023-11-16T01:51:49.151 INFO:teuthology.orchestra.run.smithi083.stdout: 2 2023-11-16T01:51:49.151 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:49.151 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_age": 1, 2023-11-16T01:51:49.151 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:51:49.151 INFO:teuthology.orchestra.run.smithi083.stdout: "required_con": "2449958747317026820", 2023-11-16T01:51:49.151 INFO:teuthology.orchestra.run.smithi083.stdout: "required_mon": [ 2023-11-16T01:51:49.151 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:51:49.152 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:51:49.152 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:51:49.152 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:51:49.152 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:51:49.152 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:51:49.152 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:51:49.152 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:51:49.153 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:49.153 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_con": "4540138314316775423", 2023-11-16T01:51:49.153 INFO:teuthology.orchestra.run.smithi083.stdout: "quorum_mon": [ 2023-11-16T01:51:49.153 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:51:49.153 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:51:49.153 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:51:49.153 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:51:49.153 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:51:49.154 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:51:49.154 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:51:49.154 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:51:49.154 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:49.154 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:49.154 INFO:teuthology.orchestra.run.smithi083.stdout: "outside_quorum": [], 2023-11-16T01:51:49.154 INFO:teuthology.orchestra.run.smithi083.stdout: "extra_probe_peers": [], 2023-11-16T01:51:49.154 INFO:teuthology.orchestra.run.smithi083.stdout: "sync_provider": [], 2023-11-16T01:51:49.155 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": { 2023-11-16T01:51:49.155 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 1, 2023-11-16T01:51:49.155 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "69ad267b-1e34-4180-8ecf-9774499b7de0", 2023-11-16T01:51:49.155 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:51:49.155 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2023-11-16T01:47:29.011826Z", 2023-11-16T01:51:49.155 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release": 16, 2023-11-16T01:51:49.155 INFO:teuthology.orchestra.run.smithi083.stdout: "min_mon_release_name": "pacific", 2023-11-16T01:51:49.155 INFO:teuthology.orchestra.run.smithi083.stdout: "election_strategy": 1, 2023-11-16T01:51:49.156 INFO:teuthology.orchestra.run.smithi083.stdout: "disallowed_leaders: ": "", 2023-11-16T01:51:49.156 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false, 2023-11-16T01:51:49.156 INFO:teuthology.orchestra.run.smithi083.stdout: "tiebreaker_mon": "", 2023-11-16T01:51:49.156 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_ranks: ": "", 2023-11-16T01:51:49.156 INFO:teuthology.orchestra.run.smithi083.stdout: "features": { 2023-11-16T01:51:49.156 INFO:teuthology.orchestra.run.smithi083.stdout: "persistent": [ 2023-11-16T01:51:49.156 INFO:teuthology.orchestra.run.smithi083.stdout: "kraken", 2023-11-16T01:51:49.156 INFO:teuthology.orchestra.run.smithi083.stdout: "luminous", 2023-11-16T01:51:49.157 INFO:teuthology.orchestra.run.smithi083.stdout: "mimic", 2023-11-16T01:51:49.157 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap-prune", 2023-11-16T01:51:49.157 INFO:teuthology.orchestra.run.smithi083.stdout: "nautilus", 2023-11-16T01:51:49.157 INFO:teuthology.orchestra.run.smithi083.stdout: "octopus", 2023-11-16T01:51:49.157 INFO:teuthology.orchestra.run.smithi083.stdout: "pacific", 2023-11-16T01:51:49.157 INFO:teuthology.orchestra.run.smithi083.stdout: "elector-pinging" 2023-11-16T01:51:49.157 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:49.157 INFO:teuthology.orchestra.run.smithi083.stdout: "optional": [] 2023-11-16T01:51:49.158 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:49.158 INFO:teuthology.orchestra.run.smithi083.stdout: "mons": [ 2023-11-16T01:51:49.158 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:49.158 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 0, 2023-11-16T01:51:49.158 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "a", 2023-11-16T01:51:49.158 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:51:49.158 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:51:49.159 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:49.159 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:51:49.159 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3300", 2023-11-16T01:51:49.159 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:49.159 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:49.159 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:49.159 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:51:49.159 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789", 2023-11-16T01:51:49.160 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:49.160 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:49.160 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:49.160 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:49.160 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6789/0", 2023-11-16T01:51:49.160 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6789/0", 2023-11-16T01:51:49.160 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:51:49.160 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:51:49.161 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:51:49.161 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:49.161 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:49.161 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 1, 2023-11-16T01:51:49.161 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "b", 2023-11-16T01:51:49.161 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:51:49.161 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:51:49.161 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:49.162 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:51:49.162 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:3300", 2023-11-16T01:51:49.162 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:49.162 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:49.162 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:49.162 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:51:49.162 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789", 2023-11-16T01:51:49.162 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:49.163 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:49.163 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:49.163 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:49.163 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.136:6789/0", 2023-11-16T01:51:49.163 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.136:6789/0", 2023-11-16T01:51:49.163 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:51:49.163 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:51:49.163 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:51:49.164 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:49.164 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:49.164 INFO:teuthology.orchestra.run.smithi083.stdout: "rank": 2, 2023-11-16T01:51:49.164 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "c", 2023-11-16T01:51:49.164 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2023-11-16T01:51:49.164 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2023-11-16T01:51:49.164 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:49.164 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v2", 2023-11-16T01:51:49.165 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:3301", 2023-11-16T01:51:49.165 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:49.165 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:49.165 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:49.165 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2023-11-16T01:51:49.165 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790", 2023-11-16T01:51:49.165 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 0 2023-11-16T01:51:49.165 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:49.166 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:49.166 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:49.166 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6790/0", 2023-11-16T01:51:49.166 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6790/0", 2023-11-16T01:51:49.166 INFO:teuthology.orchestra.run.smithi083.stdout: "priority": 0, 2023-11-16T01:51:49.166 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 0, 2023-11-16T01:51:49.166 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "{}" 2023-11-16T01:51:49.166 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:49.167 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:49.167 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:49.167 INFO:teuthology.orchestra.run.smithi083.stdout: "feature_map": { 2023-11-16T01:51:49.167 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": [ 2023-11-16T01:51:49.167 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:49.167 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:49.167 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:49.168 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:51:49.168 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:49.168 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2023-11-16T01:51:49.168 INFO:teuthology.orchestra.run.smithi083.stdout: "client": [ 2023-11-16T01:51:49.168 INFO:teuthology.orchestra.run.smithi083.stdout: { 2023-11-16T01:51:49.168 INFO:teuthology.orchestra.run.smithi083.stdout: "features": "0x3f01cfbdfffdffff", 2023-11-16T01:51:49.168 INFO:teuthology.orchestra.run.smithi083.stdout: "release": "luminous", 2023-11-16T01:51:49.169 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 1 2023-11-16T01:51:49.169 INFO:teuthology.orchestra.run.smithi083.stdout: } 2023-11-16T01:51:49.169 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2023-11-16T01:51:49.169 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2023-11-16T01:51:49.169 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": false 2023-11-16T01:51:49.169 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:51:49.170 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-16T01:51:49.170 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-11-16T01:51:49.171 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-11-16T01:51:49.514 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.514+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:51:49.514 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.514+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=10,kv=4,logm=70} crc {auth=3753404263,config=3140514009,health=1766368628,kv=3700160801,logm=3848662243}) 2023-11-16T01:51:49.514 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.514+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=12,kv=6,logm=67} crc {auth=2783635697,config=3140514009,health=3123920075,kv=3795632867,logm=3983262426}) 2023-11-16T01:51:49.515 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.514+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:51:49.515 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.514+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=10,kv=4,logm=70} crc {auth=3753404263,config=3140514009,health=1766368628,kv=3700160801,logm=3848662243}) 2023-11-16T01:51:49.515 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.514+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=11,kv=5,logm=68} crc {auth=880475573,config=3140514009,health=3340284968,kv=2868953028,logm=2529748855}) 2023-11-16T01:51:49.528 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-11-16T01:51:49.529 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.526+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:51:49.529 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.526+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1940684072}) 2023-11-16T01:51:49.529 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.526+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3635871105}) 2023-11-16T01:51:49.530 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.526+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:51:49.530 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.526+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1940684072}) 2023-11-16T01:51:49.530 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.526+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=719278760}) 2023-11-16T01:51:49.542 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.538+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:51:49.542 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.538+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=42,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=2,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=25} crc {logm=3318986376,mdsmap=1531151339,mgr=3338689344,mgr_command_descs=1448462394,mgr_metadata=2889854584,mgrstat=1405684348,mon_config_key=3373024316,monmap=1549841071,osd_metadata=2408539844,osd_pg_creating=1725079635,osdmap=1714001059}) 2023-11-16T01:51:49.542 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.542+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=42,mdsmap=3,mgr=6,mgr_metadata=1,mgrstat=10,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=25} crc {logm=4294062156,mdsmap=1531151339,mgr=3189506983,mgr_metadata=2889854584,mgrstat=558824436,mon_config_key=1707289356,monmap=1549841071,osd_metadata=2408539844,osd_pg_creating=1725079635,osdmap=4256140647}) 2023-11-16T01:51:49.542 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.542+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:51:49.543 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.542+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=42,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=2,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=25} crc {logm=3318986376,mdsmap=1531151339,mgr=3338689344,mgr_command_descs=1448462394,mgr_metadata=2889854584,mgrstat=1405684348,mon_config_key=3373024316,monmap=1549841071,osd_metadata=2408539844,osd_pg_creating=1725079635,osdmap=1714001059}) 2023-11-16T01:51:49.543 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:49.542+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=39,mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=28} crc {logm=1284406212,mdsmap=3585761405,mgr=992179106,mgr_command_descs=1448462394,mgr_metadata=2889854584,mgrstat=558824436,mon_config_key=1707289356,monmap=1549841071,osd_metadata=2408539844,osd_pg_creating=1725079635,osdmap=2588089639}) 2023-11-16T01:51:50.446 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:50.446+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:51:50.446 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:50.446+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=170440849}) 2023-11-16T01:51:50.447 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:50.446+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=83} crc {osdmap=105890433}) 2023-11-16T01:51:50.447 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:50.446+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-11-16T01:51:50.447 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:50.446+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=170440849}) 2023-11-16T01:51:50.447 INFO:tasks.ceph.mon.a.smithi083.stderr:2023-11-16T01:51:50.446+0000 7f07ee555700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=82} crc {osdmap=1015933856}) 2023-11-16T01:51:50.529 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-16T01:51:50.530 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-16T01:51:50.875 INFO:teuthology.orchestra.run.smithi083.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","modified":"2023-11-16T01:47:29.011826Z","created":"2023-11-16T01:47:29.011826Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3301","nonce":0},{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-16T01:51:50.889 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-16T01:51:50.890 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-16T01:51:50.903 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-11-16T01:51:50.903 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T01:51:51.191 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:51:51.192 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":63,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","created":"2023-11-16T01:47:58.606709+0000","modified":"2023-11-16T01:51:13.378687+0000","last_up_change":"2023-11-16T01:48:09.279683+0000","last_in_change":"2023-11-16T01:48:02.279946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":21,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-16T01:48:09.347564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-16T01:48:10.852819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9244fa2e-56f6-41bf-b5e8-264183343a6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":30853},{"type":"v1","addr":"172.21.15.83:6801","nonce":30853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":30853},{"type":"v1","addr":"172.21.15.83:6803","nonce":30853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":30853},{"type":"v1","addr":"172.21.15.83:6807","nonce":30853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":30853},{"type":"v1","addr":"172.21.15.83:6805","nonce":30853}]},"public_addr":"172.21.15.83:6801/30853","cluster_addr":"172.21.15.83:6803/30853","heartbeat_back_addr":"172.21.15.83:6807/30853","heartbeat_front_addr":"172.21.15.83:6805/30853","state":["exists","up"]},{"osd":1,"uuid":"ed5dde4d-7737-428e-af42-a56e6224529d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":30851},{"type":"v1","addr":"172.21.15.83:6809","nonce":30851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6811","nonce":30851},{"type":"v1","addr":"172.21.15.83:6813","nonce":30851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6819","nonce":30851},{"type":"v1","addr":"172.21.15.83:6821","nonce":30851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":30851},{"type":"v1","addr":"172.21.15.83:6817","nonce":30851}]},"public_addr":"172.21.15.83:6809/30851","cluster_addr":"172.21.15.83:6813/30851","heartbeat_back_addr":"172.21.15.83:6821/30851","heartbeat_front_addr":"172.21.15.83:6817/30851","state":["exists","up"]},{"osd":2,"uuid":"b363e0df-3ed6-4fd7-b06a-6fc5f880f606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":30852},{"type":"v1","addr":"172.21.15.83:6812","nonce":30852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":30852},{"type":"v1","addr":"172.21.15.83:6816","nonce":30852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":30852},{"type":"v1","addr":"172.21.15.83:6823","nonce":30852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":30852},{"type":"v1","addr":"172.21.15.83:6820","nonce":30852}]},"public_addr":"172.21.15.83:6812/30852","cluster_addr":"172.21.15.83:6816/30852","heartbeat_back_addr":"172.21.15.83:6823/30852","heartbeat_front_addr":"172.21.15.83:6820/30852","state":["exists","up"]},{"osd":3,"uuid":"a24e3b64-a290-479e-ab17-c35a891960c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":30961},{"type":"v1","addr":"172.21.15.136:6801","nonce":30961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":30961},{"type":"v1","addr":"172.21.15.136:6803","nonce":30961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":30961},{"type":"v1","addr":"172.21.15.136:6811","nonce":30961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":30961},{"type":"v1","addr":"172.21.15.136:6807","nonce":30961}]},"public_addr":"172.21.15.136:6801/30961","cluster_addr":"172.21.15.136:6803/30961","heartbeat_back_addr":"172.21.15.136:6811/30961","heartbeat_front_addr":"172.21.15.136:6807/30961","state":["exists","up"]},{"osd":4,"uuid":"f3f49f6a-7a3d-4985-a8c9-995777f7fda5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":30963},{"type":"v1","addr":"172.21.15.136:6817","nonce":30963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":30963},{"type":"v1","addr":"172.21.15.136:6819","nonce":30963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":30963},{"type":"v1","addr":"172.21.15.136:6823","nonce":30963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":30963},{"type":"v1","addr":"172.21.15.136:6821","nonce":30963}]},"public_addr":"172.21.15.136:6817/30963","cluster_addr":"172.21.15.136:6819/30963","heartbeat_back_addr":"172.21.15.136:6823/30963","heartbeat_front_addr":"172.21.15.136:6821/30963","state":["exists","up"]},{"osd":5,"uuid":"ee8f2910-b899-4798-a9d8-762ea0e7c9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":30962},{"type":"v1","addr":"172.21.15.136:6806","nonce":30962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":30962},{"type":"v1","addr":"172.21.15.136:6810","nonce":30962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":30962},{"type":"v1","addr":"172.21.15.136:6815","nonce":30962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":30962},{"type":"v1","addr":"172.21.15.136:6813","nonce":30962}]},"public_addr":"172.21.15.136:6806/30962","cluster_addr":"172.21.15.136:6810/30962","heartbeat_back_addr":"172.21.15.136:6815/30962","heartbeat_front_addr":"172.21.15.136:6813/30962","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.810281+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.823140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.962397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.892268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:08.069943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:08.018060+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T01:51:51.205 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-16T01:51:51.206 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T01:51:51.480 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:51:51.482 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2023-11-16T01:51:51.493 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2023-11-16T01:51:51.297731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786984,"kb_used_data":4408,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564444056,"statfs":{"total":579820584960,"available":577990713344,"internally_reserved":0,"allocated":4513792,"data_stored":1938348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.655978+0000","last_change":"2023-11-16T01:48:11.314302+0000","last_active":"2023-11-16T01:50:51.655978+0000","last_peered":"2023-11-16T01:50:51.655978+0000","last_clean":"2023-11-16T01:50:51.655978+0000","last_became_active":"2023-11-16T01:48:11.313884+0000","last_became_peered":"2023-11-16T01:48:11.313884+0000","last_unstale":"2023-11-16T01:50:51.655978+0000","last_undegraded":"2023-11-16T01:50:51.655978+0000","last_fullsized":"2023-11-16T01:50:51.655978+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_clean_scrub_stamp":"2023-11-16T01:48:10.292390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.597091+0000","last_change":"2023-11-16T01:48:12.316173+0000","last_active":"2023-11-16T01:50:51.597091+0000","last_peered":"2023-11-16T01:50:51.597091+0000","last_clean":"2023-11-16T01:50:51.597091+0000","last_became_active":"2023-11-16T01:48:12.315578+0000","last_became_peered":"2023-11-16T01:48:12.315578+0000","last_unstale":"2023-11-16T01:50:51.597091+0000","last_undegraded":"2023-11-16T01:50:51.597091+0000","last_fullsized":"2023-11-16T01:50:51.597091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.655825+0000","last_change":"2023-11-16T01:48:12.314951+0000","last_active":"2023-11-16T01:50:51.655825+0000","last_peered":"2023-11-16T01:50:51.655825+0000","last_clean":"2023-11-16T01:50:51.655825+0000","last_became_active":"2023-11-16T01:48:12.314325+0000","last_became_peered":"2023-11-16T01:48:12.314325+0000","last_unstale":"2023-11-16T01:50:51.655825+0000","last_undegraded":"2023-11-16T01:50:51.655825+0000","last_fullsized":"2023-11-16T01:50:51.655825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.758691+0000","last_change":"2023-11-16T01:48:12.314855+0000","last_active":"2023-11-16T01:50:51.758691+0000","last_peered":"2023-11-16T01:50:51.758691+0000","last_clean":"2023-11-16T01:50:51.758691+0000","last_became_active":"2023-11-16T01:48:12.314447+0000","last_became_peered":"2023-11-16T01:48:12.314447+0000","last_unstale":"2023-11-16T01:50:51.758691+0000","last_undegraded":"2023-11-16T01:50:51.758691+0000","last_fullsized":"2023-11-16T01:50:51.758691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.597094+0000","last_change":"2023-11-16T01:48:12.315525+0000","last_active":"2023-11-16T01:50:51.597094+0000","last_peered":"2023-11-16T01:50:51.597094+0000","last_clean":"2023-11-16T01:50:51.597094+0000","last_became_active":"2023-11-16T01:48:12.314786+0000","last_became_peered":"2023-11-16T01:48:12.314786+0000","last_unstale":"2023-11-16T01:50:51.597094+0000","last_undegraded":"2023-11-16T01:50:51.597094+0000","last_fullsized":"2023-11-16T01:50:51.597094+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.598265+0000","last_change":"2023-11-16T01:48:12.740510+0000","last_active":"2023-11-16T01:50:51.598265+0000","last_peered":"2023-11-16T01:50:51.598265+0000","last_clean":"2023-11-16T01:50:51.598265+0000","last_became_active":"2023-11-16T01:48:12.739242+0000","last_became_peered":"2023-11-16T01:48:12.739242+0000","last_unstale":"2023-11-16T01:50:51.598265+0000","last_undegraded":"2023-11-16T01:50:51.598265+0000","last_fullsized":"2023-11-16T01:50:51.598265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.655983+0000","last_change":"2023-11-16T01:48:12.740239+0000","last_active":"2023-11-16T01:50:51.655983+0000","last_peered":"2023-11-16T01:50:51.655983+0000","last_clean":"2023-11-16T01:50:51.655983+0000","last_became_active":"2023-11-16T01:48:12.739424+0000","last_became_peered":"2023-11-16T01:48:12.739424+0000","last_unstale":"2023-11-16T01:50:51.655983+0000","last_undegraded":"2023-11-16T01:50:51.655983+0000","last_fullsized":"2023-11-16T01:50:51.655983+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.598353+0000","last_change":"2023-11-16T01:48:12.315863+0000","last_active":"2023-11-16T01:50:51.598353+0000","last_peered":"2023-11-16T01:50:51.598353+0000","last_clean":"2023-11-16T01:50:51.598353+0000","last_became_active":"2023-11-16T01:48:12.315453+0000","last_became_peered":"2023-11-16T01:48:12.315453+0000","last_unstale":"2023-11-16T01:50:51.598353+0000","last_undegraded":"2023-11-16T01:50:51.598353+0000","last_fullsized":"2023-11-16T01:50:51.598353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.656031+0000","last_change":"2023-11-16T01:48:12.945654+0000","last_active":"2023-11-16T01:50:51.656031+0000","last_peered":"2023-11-16T01:50:51.656031+0000","last_clean":"2023-11-16T01:50:51.656031+0000","last_became_active":"2023-11-16T01:48:12.944985+0000","last_became_peered":"2023-11-16T01:48:12.944985+0000","last_unstale":"2023-11-16T01:50:51.656031+0000","last_undegraded":"2023-11-16T01:50:51.656031+0000","last_fullsized":"2023-11-16T01:50:51.656031+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297828,"kb_used_data":732,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074012,"statfs":{"total":96636764160,"available":96331788288,"internally_reserved":0,"allocated":749568,"data_stored":322412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73499999999999999,"15min":0.71899999999999997},"min":{"1min":0.57199999999999995,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.95899999999999996,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.75600000000000001,"15min":0.73499999999999999},"min":{"1min":0.66300000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.65600000000000003}]},{"osd":1,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.71999999999999997,"15min":0.70399999999999996},"min":{"1min":0.56599999999999995,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.89700000000000002,"5min":1.085,"15min":1.085},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.77000000000000002,"15min":0.76200000000000001},"min":{"1min":0.71299999999999997,"5min":0.317,"15min":0.317},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.86399999999999999}]},{"osd":2,"last update":"Thu Nov 16 01:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.755,"15min":0.72699999999999998},"min":{"1min":0.60199999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0149999999999999,"5min":1.369,"15min":1.369},"last":1.002},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.82799999999999996,"15min":0.80700000000000005},"min":{"1min":0.74199999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97699999999999998,"5min":1.302,"15min":1.302},"last":0.92800000000000005}]},{"osd":3,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63600000000000001,"15min":0.59999999999999998},"min":{"1min":0.47199999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.96999999999999997,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.69099999999999995,"15min":0.67600000000000005},"min":{"1min":0.53100000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0629999999999999,"5min":2.339,"15min":2.339},"last":0.80300000000000005}]},{"osd":4,"last update":"Thu Nov 16 01:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.747,"15min":0.748},"min":{"1min":0.54200000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.873,"5min":1.056,"15min":1.056},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.80600000000000005,"15min":0.79400000000000004},"min":{"1min":0.63,"5min":0.437,"15min":0.437},"max":{"1min":1.1120000000000001,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.97699999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640302,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.69999999999999996,"15min":0.67400000000000004},"min":{"1min":0.53000000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71099999999999997,"15min":0.69799999999999995},"min":{"1min":0.56599999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.0249999999999999,"5min":1.052,"15min":1.052},"last":0.77000000000000002}]},{"osd":1,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79400000000000004,"15min":0.78300000000000003},"min":{"1min":0.66300000000000003,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.96899999999999997,"5min":1.145,"15min":1.145},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.83999999999999997,"15min":0.82599999999999996},"min":{"1min":0.69999999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.024,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":1.0009999999999999}]},{"osd":2,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.75900000000000001,"15min":0.74099999999999999},"min":{"1min":0.628,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.97699999999999998,"5min":1.163,"15min":1.163},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.751,"15min":0.74099999999999999},"min":{"1min":0.60399999999999998,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.94299999999999995,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.75600000000000001}]},{"osd":3,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.66200000000000003,"15min":0.66000000000000003},"min":{"1min":0.434,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89700000000000002,"5min":1.087,"15min":1.087},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.72599999999999998,"15min":0.71999999999999997},"min":{"1min":0.56699999999999995,"5min":0.313,"15min":0.313},"max":{"1min":0.85499999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.78700000000000003}]},{"osd":5,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73099999999999998,"15min":0.72399999999999998},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0069999999999999,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.80200000000000005,"15min":0.78800000000000003},"min":{"1min":0.60199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.2769999999999999,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":1.0429999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640302,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70999999999999996,"15min":0.68300000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90300000000000002,"5min":1.014,"15min":1.014},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.75800000000000001,"15min":0.72999999999999998},"min":{"1min":0.52500000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.86299999999999999}]},{"osd":1,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.749,"15min":0.73099999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0289999999999999,"5min":1.129,"15min":1.129},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.755,"5min":0.75800000000000001,"15min":0.74199999999999999},"min":{"1min":0.59399999999999997,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.97699999999999998,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.83399999999999996}]},{"osd":2,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.745,"15min":0.72099999999999997},"min":{"1min":0.59499999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.093,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.81499999999999995,"15min":0.78400000000000003},"min":{"1min":0.60099999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.90600000000000003}]},{"osd":4,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65200000000000002,"15min":0.64000000000000001},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89600000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.54700000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.98399999999999999,"5min":1.218,"15min":1.218},"last":0.63}]},{"osd":5,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.78400000000000003,"15min":0.745},"min":{"1min":0.56699999999999995,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.85099999999999998,"15min":0.83599999999999997},"min":{"1min":0.70699999999999996,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.079,"5min":1.409,"15min":1.409},"last":0.73199999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.47199999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.65500000000000003,"5min":0.747,"15min":0.747},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57499999999999996,"15min":0.56200000000000006},"min":{"1min":0.48299999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.61399999999999999}]},{"osd":1,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.68500000000000005,"15min":0.69399999999999995},"min":{"1min":0.44600000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.92800000000000005,"5min":0.997,"15min":0.997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.69699999999999995,"15min":0.69299999999999995},"min":{"1min":0.503,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.67200000000000004}]},{"osd":3,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.78500000000000003,"15min":0.73199999999999998},"min":{"1min":0.63900000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.1419999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86499999999999999,"15min":0.84299999999999997},"min":{"1min":0.71399999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.016,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.90900000000000003}]},{"osd":4,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84899999999999998,"15min":0.83299999999999996},"min":{"1min":0.626,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.3280000000000001,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.82699999999999996,"15min":0.80300000000000005},"min":{"1min":0.64100000000000001,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.3,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.80900000000000005}]},{"osd":5,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.78400000000000003,"15min":0.754},"min":{"1min":0.67400000000000004,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.84399999999999997,"15min":0.81599999999999995},"min":{"1min":0.70799999999999996,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.76100000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640303,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297828,"kb_used_data":732,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074012,"statfs":{"total":96636764160,"available":96331788288,"internally_reserved":0,"allocated":749568,"data_stored":322412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71599999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.64700000000000002,"15min":0.63500000000000001},"min":{"1min":0.50900000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.70599999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.67800000000000005}]},{"osd":3,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.80200000000000005,"15min":0.77600000000000002},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96399999999999997,"5min":1.127,"15min":1.127},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.879,"5min":0.83099999999999996,"15min":0.80900000000000005},"min":{"1min":0.76700000000000002,"5min":0.62,"15min":0.62},"max":{"1min":1.1379999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.79100000000000004}]},{"osd":4,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.80800000000000005,"15min":0.80200000000000005},"min":{"1min":0.65600000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.98199999999999998,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.84199999999999997,"15min":0.83199999999999996},"min":{"1min":0.67300000000000004,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.84899999999999998}]},{"osd":5,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.82599999999999996,"15min":0.81399999999999995},"min":{"1min":0.69799999999999995,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.85799999999999998,"15min":0.84299999999999997},"min":{"1min":0.73999999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.1080000000000001,"5min":1.403,"15min":1.403},"last":0.77300000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640304,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.47599999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.90200000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.47899999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.496}]},{"osd":2,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.52700000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.93500000000000005,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.44900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.95499999999999996,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.61099999999999999}]},{"osd":3,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.81100000000000005,"15min":0.78100000000000003},"min":{"1min":0.70699999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.84699999999999998,"15min":0.82699999999999996},"min":{"1min":0.71499999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.88}]},{"osd":4,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79000000000000004},"min":{"1min":0.61299999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0349999999999999,"5min":1.242,"15min":1.242},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.79800000000000004,"15min":0.76400000000000001},"min":{"1min":0.73999999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.81699999999999995}]},{"osd":5,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.85199999999999998,"15min":0.83799999999999997},"min":{"1min":0.71399999999999997,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1419999999999999,"5min":1.147,"15min":1.147},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.86799999999999999,"15min":0.85199999999999998},"min":{"1min":0.71599999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.032,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.90000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T01:51:51.495 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T01:51:51.772 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:51:51.774 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2023-11-16T01:51:51.786 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2023-11-16T01:51:51.297731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786984,"kb_used_data":4408,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564444056,"statfs":{"total":579820584960,"available":577990713344,"internally_reserved":0,"allocated":4513792,"data_stored":1938348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.655978+0000","last_change":"2023-11-16T01:48:11.314302+0000","last_active":"2023-11-16T01:50:51.655978+0000","last_peered":"2023-11-16T01:50:51.655978+0000","last_clean":"2023-11-16T01:50:51.655978+0000","last_became_active":"2023-11-16T01:48:11.313884+0000","last_became_peered":"2023-11-16T01:48:11.313884+0000","last_unstale":"2023-11-16T01:50:51.655978+0000","last_undegraded":"2023-11-16T01:50:51.655978+0000","last_fullsized":"2023-11-16T01:50:51.655978+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_clean_scrub_stamp":"2023-11-16T01:48:10.292390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.597091+0000","last_change":"2023-11-16T01:48:12.316173+0000","last_active":"2023-11-16T01:50:51.597091+0000","last_peered":"2023-11-16T01:50:51.597091+0000","last_clean":"2023-11-16T01:50:51.597091+0000","last_became_active":"2023-11-16T01:48:12.315578+0000","last_became_peered":"2023-11-16T01:48:12.315578+0000","last_unstale":"2023-11-16T01:50:51.597091+0000","last_undegraded":"2023-11-16T01:50:51.597091+0000","last_fullsized":"2023-11-16T01:50:51.597091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.655825+0000","last_change":"2023-11-16T01:48:12.314951+0000","last_active":"2023-11-16T01:50:51.655825+0000","last_peered":"2023-11-16T01:50:51.655825+0000","last_clean":"2023-11-16T01:50:51.655825+0000","last_became_active":"2023-11-16T01:48:12.314325+0000","last_became_peered":"2023-11-16T01:48:12.314325+0000","last_unstale":"2023-11-16T01:50:51.655825+0000","last_undegraded":"2023-11-16T01:50:51.655825+0000","last_fullsized":"2023-11-16T01:50:51.655825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.758691+0000","last_change":"2023-11-16T01:48:12.314855+0000","last_active":"2023-11-16T01:50:51.758691+0000","last_peered":"2023-11-16T01:50:51.758691+0000","last_clean":"2023-11-16T01:50:51.758691+0000","last_became_active":"2023-11-16T01:48:12.314447+0000","last_became_peered":"2023-11-16T01:48:12.314447+0000","last_unstale":"2023-11-16T01:50:51.758691+0000","last_undegraded":"2023-11-16T01:50:51.758691+0000","last_fullsized":"2023-11-16T01:50:51.758691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.597094+0000","last_change":"2023-11-16T01:48:12.315525+0000","last_active":"2023-11-16T01:50:51.597094+0000","last_peered":"2023-11-16T01:50:51.597094+0000","last_clean":"2023-11-16T01:50:51.597094+0000","last_became_active":"2023-11-16T01:48:12.314786+0000","last_became_peered":"2023-11-16T01:48:12.314786+0000","last_unstale":"2023-11-16T01:50:51.597094+0000","last_undegraded":"2023-11-16T01:50:51.597094+0000","last_fullsized":"2023-11-16T01:50:51.597094+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.598265+0000","last_change":"2023-11-16T01:48:12.740510+0000","last_active":"2023-11-16T01:50:51.598265+0000","last_peered":"2023-11-16T01:50:51.598265+0000","last_clean":"2023-11-16T01:50:51.598265+0000","last_became_active":"2023-11-16T01:48:12.739242+0000","last_became_peered":"2023-11-16T01:48:12.739242+0000","last_unstale":"2023-11-16T01:50:51.598265+0000","last_undegraded":"2023-11-16T01:50:51.598265+0000","last_fullsized":"2023-11-16T01:50:51.598265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.655983+0000","last_change":"2023-11-16T01:48:12.740239+0000","last_active":"2023-11-16T01:50:51.655983+0000","last_peered":"2023-11-16T01:50:51.655983+0000","last_clean":"2023-11-16T01:50:51.655983+0000","last_became_active":"2023-11-16T01:48:12.739424+0000","last_became_peered":"2023-11-16T01:48:12.739424+0000","last_unstale":"2023-11-16T01:50:51.655983+0000","last_undegraded":"2023-11-16T01:50:51.655983+0000","last_fullsized":"2023-11-16T01:50:51.655983+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.598353+0000","last_change":"2023-11-16T01:48:12.315863+0000","last_active":"2023-11-16T01:50:51.598353+0000","last_peered":"2023-11-16T01:50:51.598353+0000","last_clean":"2023-11-16T01:50:51.598353+0000","last_became_active":"2023-11-16T01:48:12.315453+0000","last_became_peered":"2023-11-16T01:48:12.315453+0000","last_unstale":"2023-11-16T01:50:51.598353+0000","last_undegraded":"2023-11-16T01:50:51.598353+0000","last_fullsized":"2023-11-16T01:50:51.598353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.656031+0000","last_change":"2023-11-16T01:48:12.945654+0000","last_active":"2023-11-16T01:50:51.656031+0000","last_peered":"2023-11-16T01:50:51.656031+0000","last_clean":"2023-11-16T01:50:51.656031+0000","last_became_active":"2023-11-16T01:48:12.944985+0000","last_became_peered":"2023-11-16T01:48:12.944985+0000","last_unstale":"2023-11-16T01:50:51.656031+0000","last_undegraded":"2023-11-16T01:50:51.656031+0000","last_fullsized":"2023-11-16T01:50:51.656031+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297828,"kb_used_data":732,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074012,"statfs":{"total":96636764160,"available":96331788288,"internally_reserved":0,"allocated":749568,"data_stored":322412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73499999999999999,"15min":0.71899999999999997},"min":{"1min":0.57199999999999995,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.95899999999999996,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.75600000000000001,"15min":0.73499999999999999},"min":{"1min":0.66300000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.65600000000000003}]},{"osd":1,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.71999999999999997,"15min":0.70399999999999996},"min":{"1min":0.56599999999999995,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.89700000000000002,"5min":1.085,"15min":1.085},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.77000000000000002,"15min":0.76200000000000001},"min":{"1min":0.71299999999999997,"5min":0.317,"15min":0.317},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.86399999999999999}]},{"osd":2,"last update":"Thu Nov 16 01:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.755,"15min":0.72699999999999998},"min":{"1min":0.60199999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0149999999999999,"5min":1.369,"15min":1.369},"last":1.002},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.82799999999999996,"15min":0.80700000000000005},"min":{"1min":0.74199999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97699999999999998,"5min":1.302,"15min":1.302},"last":0.92800000000000005}]},{"osd":3,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63600000000000001,"15min":0.59999999999999998},"min":{"1min":0.47199999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.96999999999999997,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.69099999999999995,"15min":0.67600000000000005},"min":{"1min":0.53100000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0629999999999999,"5min":2.339,"15min":2.339},"last":0.80300000000000005}]},{"osd":4,"last update":"Thu Nov 16 01:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.747,"15min":0.748},"min":{"1min":0.54200000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.873,"5min":1.056,"15min":1.056},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.80600000000000005,"15min":0.79400000000000004},"min":{"1min":0.63,"5min":0.437,"15min":0.437},"max":{"1min":1.1120000000000001,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.97699999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640302,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.69999999999999996,"15min":0.67400000000000004},"min":{"1min":0.53000000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71099999999999997,"15min":0.69799999999999995},"min":{"1min":0.56599999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.0249999999999999,"5min":1.052,"15min":1.052},"last":0.77000000000000002}]},{"osd":1,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79400000000000004,"15min":0.78300000000000003},"min":{"1min":0.66300000000000003,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.96899999999999997,"5min":1.145,"15min":1.145},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.83999999999999997,"15min":0.82599999999999996},"min":{"1min":0.69999999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.024,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":1.0009999999999999}]},{"osd":2,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.75900000000000001,"15min":0.74099999999999999},"min":{"1min":0.628,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.97699999999999998,"5min":1.163,"15min":1.163},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.751,"15min":0.74099999999999999},"min":{"1min":0.60399999999999998,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.94299999999999995,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.75600000000000001}]},{"osd":3,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.66200000000000003,"15min":0.66000000000000003},"min":{"1min":0.434,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89700000000000002,"5min":1.087,"15min":1.087},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.72599999999999998,"15min":0.71999999999999997},"min":{"1min":0.56699999999999995,"5min":0.313,"15min":0.313},"max":{"1min":0.85499999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.78700000000000003}]},{"osd":5,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73099999999999998,"15min":0.72399999999999998},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0069999999999999,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.80200000000000005,"15min":0.78800000000000003},"min":{"1min":0.60199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.2769999999999999,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":1.0429999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640302,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70999999999999996,"15min":0.68300000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90300000000000002,"5min":1.014,"15min":1.014},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.75800000000000001,"15min":0.72999999999999998},"min":{"1min":0.52500000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.86299999999999999}]},{"osd":1,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.749,"15min":0.73099999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0289999999999999,"5min":1.129,"15min":1.129},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.755,"5min":0.75800000000000001,"15min":0.74199999999999999},"min":{"1min":0.59399999999999997,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.97699999999999998,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.83399999999999996}]},{"osd":2,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.745,"15min":0.72099999999999997},"min":{"1min":0.59499999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.093,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.81499999999999995,"15min":0.78400000000000003},"min":{"1min":0.60099999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.90600000000000003}]},{"osd":4,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65200000000000002,"15min":0.64000000000000001},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89600000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.54700000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.98399999999999999,"5min":1.218,"15min":1.218},"last":0.63}]},{"osd":5,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.78400000000000003,"15min":0.745},"min":{"1min":0.56699999999999995,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.85099999999999998,"15min":0.83599999999999997},"min":{"1min":0.70699999999999996,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.079,"5min":1.409,"15min":1.409},"last":0.73199999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.47199999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.65500000000000003,"5min":0.747,"15min":0.747},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57499999999999996,"15min":0.56200000000000006},"min":{"1min":0.48299999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.61399999999999999}]},{"osd":1,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.68500000000000005,"15min":0.69399999999999995},"min":{"1min":0.44600000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.92800000000000005,"5min":0.997,"15min":0.997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.69699999999999995,"15min":0.69299999999999995},"min":{"1min":0.503,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.67200000000000004}]},{"osd":3,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.78500000000000003,"15min":0.73199999999999998},"min":{"1min":0.63900000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.1419999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86499999999999999,"15min":0.84299999999999997},"min":{"1min":0.71399999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.016,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.90900000000000003}]},{"osd":4,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84899999999999998,"15min":0.83299999999999996},"min":{"1min":0.626,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.3280000000000001,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.82699999999999996,"15min":0.80300000000000005},"min":{"1min":0.64100000000000001,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.3,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.80900000000000005}]},{"osd":5,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.78400000000000003,"15min":0.754},"min":{"1min":0.67400000000000004,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.84399999999999997,"15min":0.81599999999999995},"min":{"1min":0.70799999999999996,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.76100000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640303,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297828,"kb_used_data":732,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074012,"statfs":{"total":96636764160,"available":96331788288,"internally_reserved":0,"allocated":749568,"data_stored":322412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71599999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.64700000000000002,"15min":0.63500000000000001},"min":{"1min":0.50900000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.70599999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.67800000000000005}]},{"osd":3,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.80200000000000005,"15min":0.77600000000000002},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96399999999999997,"5min":1.127,"15min":1.127},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.879,"5min":0.83099999999999996,"15min":0.80900000000000005},"min":{"1min":0.76700000000000002,"5min":0.62,"15min":0.62},"max":{"1min":1.1379999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.79100000000000004}]},{"osd":4,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.80800000000000005,"15min":0.80200000000000005},"min":{"1min":0.65600000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.98199999999999998,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.84199999999999997,"15min":0.83199999999999996},"min":{"1min":0.67300000000000004,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.84899999999999998}]},{"osd":5,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.82599999999999996,"15min":0.81399999999999995},"min":{"1min":0.69799999999999995,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.85799999999999998,"15min":0.84299999999999997},"min":{"1min":0.73999999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.1080000000000001,"5min":1.403,"15min":1.403},"last":0.77300000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640304,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.47599999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.90200000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.47899999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.496}]},{"osd":2,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.52700000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.93500000000000005,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.44900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.95499999999999996,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.61099999999999999}]},{"osd":3,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.81100000000000005,"15min":0.78100000000000003},"min":{"1min":0.70699999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.84699999999999998,"15min":0.82699999999999996},"min":{"1min":0.71499999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.88}]},{"osd":4,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79000000000000004},"min":{"1min":0.61299999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0349999999999999,"5min":1.242,"15min":1.242},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.79800000000000004,"15min":0.76400000000000001},"min":{"1min":0.73999999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.81699999999999995}]},{"osd":5,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.85199999999999998,"15min":0.83799999999999997},"min":{"1min":0.71399999999999997,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1419999999999999,"5min":1.147,"15min":1.147},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.86799999999999999,"15min":0.85199999999999998},"min":{"1min":0.71599999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.032,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.90000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T01:51:51.787 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-16T01:51:51.788 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T01:51:52.069 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:51:52.072 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2023-11-16T01:51:52.083 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2023-11-16T01:51:51.297731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786984,"kb_used_data":4408,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564444056,"statfs":{"total":579820584960,"available":577990713344,"internally_reserved":0,"allocated":4513792,"data_stored":1938348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.655978+0000","last_change":"2023-11-16T01:48:11.314302+0000","last_active":"2023-11-16T01:50:51.655978+0000","last_peered":"2023-11-16T01:50:51.655978+0000","last_clean":"2023-11-16T01:50:51.655978+0000","last_became_active":"2023-11-16T01:48:11.313884+0000","last_became_peered":"2023-11-16T01:48:11.313884+0000","last_unstale":"2023-11-16T01:50:51.655978+0000","last_undegraded":"2023-11-16T01:50:51.655978+0000","last_fullsized":"2023-11-16T01:50:51.655978+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_clean_scrub_stamp":"2023-11-16T01:48:10.292390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.597091+0000","last_change":"2023-11-16T01:48:12.316173+0000","last_active":"2023-11-16T01:50:51.597091+0000","last_peered":"2023-11-16T01:50:51.597091+0000","last_clean":"2023-11-16T01:50:51.597091+0000","last_became_active":"2023-11-16T01:48:12.315578+0000","last_became_peered":"2023-11-16T01:48:12.315578+0000","last_unstale":"2023-11-16T01:50:51.597091+0000","last_undegraded":"2023-11-16T01:50:51.597091+0000","last_fullsized":"2023-11-16T01:50:51.597091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.655825+0000","last_change":"2023-11-16T01:48:12.314951+0000","last_active":"2023-11-16T01:50:51.655825+0000","last_peered":"2023-11-16T01:50:51.655825+0000","last_clean":"2023-11-16T01:50:51.655825+0000","last_became_active":"2023-11-16T01:48:12.314325+0000","last_became_peered":"2023-11-16T01:48:12.314325+0000","last_unstale":"2023-11-16T01:50:51.655825+0000","last_undegraded":"2023-11-16T01:50:51.655825+0000","last_fullsized":"2023-11-16T01:50:51.655825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.758691+0000","last_change":"2023-11-16T01:48:12.314855+0000","last_active":"2023-11-16T01:50:51.758691+0000","last_peered":"2023-11-16T01:50:51.758691+0000","last_clean":"2023-11-16T01:50:51.758691+0000","last_became_active":"2023-11-16T01:48:12.314447+0000","last_became_peered":"2023-11-16T01:48:12.314447+0000","last_unstale":"2023-11-16T01:50:51.758691+0000","last_undegraded":"2023-11-16T01:50:51.758691+0000","last_fullsized":"2023-11-16T01:50:51.758691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.597094+0000","last_change":"2023-11-16T01:48:12.315525+0000","last_active":"2023-11-16T01:50:51.597094+0000","last_peered":"2023-11-16T01:50:51.597094+0000","last_clean":"2023-11-16T01:50:51.597094+0000","last_became_active":"2023-11-16T01:48:12.314786+0000","last_became_peered":"2023-11-16T01:48:12.314786+0000","last_unstale":"2023-11-16T01:50:51.597094+0000","last_undegraded":"2023-11-16T01:50:51.597094+0000","last_fullsized":"2023-11-16T01:50:51.597094+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.598265+0000","last_change":"2023-11-16T01:48:12.740510+0000","last_active":"2023-11-16T01:50:51.598265+0000","last_peered":"2023-11-16T01:50:51.598265+0000","last_clean":"2023-11-16T01:50:51.598265+0000","last_became_active":"2023-11-16T01:48:12.739242+0000","last_became_peered":"2023-11-16T01:48:12.739242+0000","last_unstale":"2023-11-16T01:50:51.598265+0000","last_undegraded":"2023-11-16T01:50:51.598265+0000","last_fullsized":"2023-11-16T01:50:51.598265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.655983+0000","last_change":"2023-11-16T01:48:12.740239+0000","last_active":"2023-11-16T01:50:51.655983+0000","last_peered":"2023-11-16T01:50:51.655983+0000","last_clean":"2023-11-16T01:50:51.655983+0000","last_became_active":"2023-11-16T01:48:12.739424+0000","last_became_peered":"2023-11-16T01:48:12.739424+0000","last_unstale":"2023-11-16T01:50:51.655983+0000","last_undegraded":"2023-11-16T01:50:51.655983+0000","last_fullsized":"2023-11-16T01:50:51.655983+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.598353+0000","last_change":"2023-11-16T01:48:12.315863+0000","last_active":"2023-11-16T01:50:51.598353+0000","last_peered":"2023-11-16T01:50:51.598353+0000","last_clean":"2023-11-16T01:50:51.598353+0000","last_became_active":"2023-11-16T01:48:12.315453+0000","last_became_peered":"2023-11-16T01:48:12.315453+0000","last_unstale":"2023-11-16T01:50:51.598353+0000","last_undegraded":"2023-11-16T01:50:51.598353+0000","last_fullsized":"2023-11-16T01:50:51.598353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.656031+0000","last_change":"2023-11-16T01:48:12.945654+0000","last_active":"2023-11-16T01:50:51.656031+0000","last_peered":"2023-11-16T01:50:51.656031+0000","last_clean":"2023-11-16T01:50:51.656031+0000","last_became_active":"2023-11-16T01:48:12.944985+0000","last_became_peered":"2023-11-16T01:48:12.944985+0000","last_unstale":"2023-11-16T01:50:51.656031+0000","last_undegraded":"2023-11-16T01:50:51.656031+0000","last_fullsized":"2023-11-16T01:50:51.656031+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297828,"kb_used_data":732,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074012,"statfs":{"total":96636764160,"available":96331788288,"internally_reserved":0,"allocated":749568,"data_stored":322412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73499999999999999,"15min":0.71899999999999997},"min":{"1min":0.57199999999999995,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.95899999999999996,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.75600000000000001,"15min":0.73499999999999999},"min":{"1min":0.66300000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.65600000000000003}]},{"osd":1,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.71999999999999997,"15min":0.70399999999999996},"min":{"1min":0.56599999999999995,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.89700000000000002,"5min":1.085,"15min":1.085},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.77000000000000002,"15min":0.76200000000000001},"min":{"1min":0.71299999999999997,"5min":0.317,"15min":0.317},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.86399999999999999}]},{"osd":2,"last update":"Thu Nov 16 01:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.755,"15min":0.72699999999999998},"min":{"1min":0.60199999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0149999999999999,"5min":1.369,"15min":1.369},"last":1.002},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.82799999999999996,"15min":0.80700000000000005},"min":{"1min":0.74199999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97699999999999998,"5min":1.302,"15min":1.302},"last":0.92800000000000005}]},{"osd":3,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63600000000000001,"15min":0.59999999999999998},"min":{"1min":0.47199999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.96999999999999997,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.69099999999999995,"15min":0.67600000000000005},"min":{"1min":0.53100000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0629999999999999,"5min":2.339,"15min":2.339},"last":0.80300000000000005}]},{"osd":4,"last update":"Thu Nov 16 01:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.747,"15min":0.748},"min":{"1min":0.54200000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.873,"5min":1.056,"15min":1.056},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.80600000000000005,"15min":0.79400000000000004},"min":{"1min":0.63,"5min":0.437,"15min":0.437},"max":{"1min":1.1120000000000001,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.97699999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640302,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.69999999999999996,"15min":0.67400000000000004},"min":{"1min":0.53000000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71099999999999997,"15min":0.69799999999999995},"min":{"1min":0.56599999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.0249999999999999,"5min":1.052,"15min":1.052},"last":0.77000000000000002}]},{"osd":1,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79400000000000004,"15min":0.78300000000000003},"min":{"1min":0.66300000000000003,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.96899999999999997,"5min":1.145,"15min":1.145},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.83999999999999997,"15min":0.82599999999999996},"min":{"1min":0.69999999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.024,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":1.0009999999999999}]},{"osd":2,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.75900000000000001,"15min":0.74099999999999999},"min":{"1min":0.628,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.97699999999999998,"5min":1.163,"15min":1.163},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.751,"15min":0.74099999999999999},"min":{"1min":0.60399999999999998,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.94299999999999995,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.75600000000000001}]},{"osd":3,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.66200000000000003,"15min":0.66000000000000003},"min":{"1min":0.434,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89700000000000002,"5min":1.087,"15min":1.087},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.72599999999999998,"15min":0.71999999999999997},"min":{"1min":0.56699999999999995,"5min":0.313,"15min":0.313},"max":{"1min":0.85499999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.78700000000000003}]},{"osd":5,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73099999999999998,"15min":0.72399999999999998},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0069999999999999,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.80200000000000005,"15min":0.78800000000000003},"min":{"1min":0.60199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.2769999999999999,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":1.0429999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640302,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70999999999999996,"15min":0.68300000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90300000000000002,"5min":1.014,"15min":1.014},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.75800000000000001,"15min":0.72999999999999998},"min":{"1min":0.52500000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.86299999999999999}]},{"osd":1,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.749,"15min":0.73099999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0289999999999999,"5min":1.129,"15min":1.129},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.755,"5min":0.75800000000000001,"15min":0.74199999999999999},"min":{"1min":0.59399999999999997,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.97699999999999998,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.83399999999999996}]},{"osd":2,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.745,"15min":0.72099999999999997},"min":{"1min":0.59499999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.093,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.81499999999999995,"15min":0.78400000000000003},"min":{"1min":0.60099999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.90600000000000003}]},{"osd":4,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65200000000000002,"15min":0.64000000000000001},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89600000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.54700000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.98399999999999999,"5min":1.218,"15min":1.218},"last":0.63}]},{"osd":5,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.78400000000000003,"15min":0.745},"min":{"1min":0.56699999999999995,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.85099999999999998,"15min":0.83599999999999997},"min":{"1min":0.70699999999999996,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.079,"5min":1.409,"15min":1.409},"last":0.73199999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.47199999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.65500000000000003,"5min":0.747,"15min":0.747},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57499999999999996,"15min":0.56200000000000006},"min":{"1min":0.48299999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.61399999999999999}]},{"osd":1,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.68500000000000005,"15min":0.69399999999999995},"min":{"1min":0.44600000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.92800000000000005,"5min":0.997,"15min":0.997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.69699999999999995,"15min":0.69299999999999995},"min":{"1min":0.503,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.67200000000000004}]},{"osd":3,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.78500000000000003,"15min":0.73199999999999998},"min":{"1min":0.63900000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.1419999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86499999999999999,"15min":0.84299999999999997},"min":{"1min":0.71399999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.016,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.90900000000000003}]},{"osd":4,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84899999999999998,"15min":0.83299999999999996},"min":{"1min":0.626,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.3280000000000001,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.82699999999999996,"15min":0.80300000000000005},"min":{"1min":0.64100000000000001,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.3,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.80900000000000005}]},{"osd":5,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.78400000000000003,"15min":0.754},"min":{"1min":0.67400000000000004,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.84399999999999997,"15min":0.81599999999999995},"min":{"1min":0.70799999999999996,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.76100000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640303,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297828,"kb_used_data":732,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074012,"statfs":{"total":96636764160,"available":96331788288,"internally_reserved":0,"allocated":749568,"data_stored":322412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71599999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.64700000000000002,"15min":0.63500000000000001},"min":{"1min":0.50900000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.70599999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.67800000000000005}]},{"osd":3,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.80200000000000005,"15min":0.77600000000000002},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96399999999999997,"5min":1.127,"15min":1.127},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.879,"5min":0.83099999999999996,"15min":0.80900000000000005},"min":{"1min":0.76700000000000002,"5min":0.62,"15min":0.62},"max":{"1min":1.1379999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.79100000000000004}]},{"osd":4,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.80800000000000005,"15min":0.80200000000000005},"min":{"1min":0.65600000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.98199999999999998,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.84199999999999997,"15min":0.83199999999999996},"min":{"1min":0.67300000000000004,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.84899999999999998}]},{"osd":5,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.82599999999999996,"15min":0.81399999999999995},"min":{"1min":0.69799999999999995,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.85799999999999998,"15min":0.84299999999999997},"min":{"1min":0.73999999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.1080000000000001,"5min":1.403,"15min":1.403},"last":0.77300000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640304,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.47599999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.90200000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.47899999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.496}]},{"osd":2,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.52700000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.93500000000000005,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.44900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.95499999999999996,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.61099999999999999}]},{"osd":3,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.81100000000000005,"15min":0.78100000000000003},"min":{"1min":0.70699999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.84699999999999998,"15min":0.82699999999999996},"min":{"1min":0.71499999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.88}]},{"osd":4,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79000000000000004},"min":{"1min":0.61299999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0349999999999999,"5min":1.242,"15min":1.242},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.79800000000000004,"15min":0.76400000000000001},"min":{"1min":0.73999999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.81699999999999995}]},{"osd":5,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.85199999999999998,"15min":0.83799999999999997},"min":{"1min":0.71399999999999997,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1419999999999999,"5min":1.147,"15min":1.147},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.86799999999999999,"15min":0.85199999999999998},"min":{"1min":0.71599999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.032,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.90000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T01:51:52.084 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T01:51:52.363 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:51:52.363 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":63,"fsid":"69ad267b-1e34-4180-8ecf-9774499b7de0","created":"2023-11-16T01:47:58.606709+0000","modified":"2023-11-16T01:51:13.378687+0000","last_up_change":"2023-11-16T01:48:09.279683+0000","last_in_change":"2023-11-16T01:48:02.279946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":21,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-16T01:48:09.347564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-16T01:48:10.852819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9244fa2e-56f6-41bf-b5e8-264183343a6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":30853},{"type":"v1","addr":"172.21.15.83:6801","nonce":30853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":30853},{"type":"v1","addr":"172.21.15.83:6803","nonce":30853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":30853},{"type":"v1","addr":"172.21.15.83:6807","nonce":30853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":30853},{"type":"v1","addr":"172.21.15.83:6805","nonce":30853}]},"public_addr":"172.21.15.83:6801/30853","cluster_addr":"172.21.15.83:6803/30853","heartbeat_back_addr":"172.21.15.83:6807/30853","heartbeat_front_addr":"172.21.15.83:6805/30853","state":["exists","up"]},{"osd":1,"uuid":"ed5dde4d-7737-428e-af42-a56e6224529d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":30851},{"type":"v1","addr":"172.21.15.83:6809","nonce":30851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6811","nonce":30851},{"type":"v1","addr":"172.21.15.83:6813","nonce":30851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6819","nonce":30851},{"type":"v1","addr":"172.21.15.83:6821","nonce":30851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":30851},{"type":"v1","addr":"172.21.15.83:6817","nonce":30851}]},"public_addr":"172.21.15.83:6809/30851","cluster_addr":"172.21.15.83:6813/30851","heartbeat_back_addr":"172.21.15.83:6821/30851","heartbeat_front_addr":"172.21.15.83:6817/30851","state":["exists","up"]},{"osd":2,"uuid":"b363e0df-3ed6-4fd7-b06a-6fc5f880f606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":30852},{"type":"v1","addr":"172.21.15.83:6812","nonce":30852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":30852},{"type":"v1","addr":"172.21.15.83:6816","nonce":30852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":30852},{"type":"v1","addr":"172.21.15.83:6823","nonce":30852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":30852},{"type":"v1","addr":"172.21.15.83:6820","nonce":30852}]},"public_addr":"172.21.15.83:6812/30852","cluster_addr":"172.21.15.83:6816/30852","heartbeat_back_addr":"172.21.15.83:6823/30852","heartbeat_front_addr":"172.21.15.83:6820/30852","state":["exists","up"]},{"osd":3,"uuid":"a24e3b64-a290-479e-ab17-c35a891960c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":30961},{"type":"v1","addr":"172.21.15.136:6801","nonce":30961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":30961},{"type":"v1","addr":"172.21.15.136:6803","nonce":30961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":30961},{"type":"v1","addr":"172.21.15.136:6811","nonce":30961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":30961},{"type":"v1","addr":"172.21.15.136:6807","nonce":30961}]},"public_addr":"172.21.15.136:6801/30961","cluster_addr":"172.21.15.136:6803/30961","heartbeat_back_addr":"172.21.15.136:6811/30961","heartbeat_front_addr":"172.21.15.136:6807/30961","state":["exists","up"]},{"osd":4,"uuid":"f3f49f6a-7a3d-4985-a8c9-995777f7fda5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":30963},{"type":"v1","addr":"172.21.15.136:6817","nonce":30963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":30963},{"type":"v1","addr":"172.21.15.136:6819","nonce":30963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":30963},{"type":"v1","addr":"172.21.15.136:6823","nonce":30963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":30963},{"type":"v1","addr":"172.21.15.136:6821","nonce":30963}]},"public_addr":"172.21.15.136:6817/30963","cluster_addr":"172.21.15.136:6819/30963","heartbeat_back_addr":"172.21.15.136:6823/30963","heartbeat_front_addr":"172.21.15.136:6821/30963","state":["exists","up"]},{"osd":5,"uuid":"ee8f2910-b899-4798-a9d8-762ea0e7c9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":30962},{"type":"v1","addr":"172.21.15.136:6806","nonce":30962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":30962},{"type":"v1","addr":"172.21.15.136:6810","nonce":30962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":30962},{"type":"v1","addr":"172.21.15.136:6815","nonce":30962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":30962},{"type":"v1","addr":"172.21.15.136:6813","nonce":30962}]},"public_addr":"172.21.15.136:6806/30962","cluster_addr":"172.21.15.136:6810/30962","heartbeat_back_addr":"172.21.15.136:6815/30962","heartbeat_front_addr":"172.21.15.136:6813/30962","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.810281+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.823140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.962397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:07.892268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:08.069943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T01:48:08.018060+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T01:51:53.377 INFO:tasks.ceph:Scrubbing osd.0 2023-11-16T01:51:53.378 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-11-16T01:51:53.543 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:51:53.543 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-16T01:51:53.543 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:51:53.556 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-11-16T01:51:53.839 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 0 to deep-scrub 2023-11-16T01:51:53.848 INFO:tasks.ceph:Scrubbing osd.1 2023-11-16T01:51:53.849 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-11-16T01:51:53.997 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:51:53.998 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-16T01:51:53.998 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:51:54.009 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-11-16T01:51:54.296 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 1 to deep-scrub 2023-11-16T01:51:54.306 INFO:tasks.ceph:Scrubbing osd.2 2023-11-16T01:51:54.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-11-16T01:51:54.457 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:51:54.457 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-16T01:51:54.457 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:51:54.469 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-11-16T01:51:54.760 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 2 to deep-scrub 2023-11-16T01:51:54.774 INFO:tasks.ceph:Scrubbing osd.3 2023-11-16T01:51:54.775 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-11-16T01:51:54.933 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:51:54.934 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-16T01:51:54.934 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:51:54.946 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-11-16T01:51:55.237 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 3 to deep-scrub 2023-11-16T01:51:55.247 INFO:tasks.ceph:Scrubbing osd.4 2023-11-16T01:51:55.247 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-11-16T01:51:55.396 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:51:55.397 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-16T01:51:55.397 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:51:55.415 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-11-16T01:51:55.707 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 4 to deep-scrub 2023-11-16T01:51:55.717 INFO:tasks.ceph:Scrubbing osd.5 2023-11-16T01:51:55.718 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-11-16T01:51:55.875 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2023-11-16T01:51:55.876 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-16T01:51:55.876 INFO:teuthology.orchestra.run.smithi083.stdout:} 2023-11-16T01:51:55.888 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-11-16T01:51:56.179 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 5 to deep-scrub 2023-11-16T01:51:56.189 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T01:51:56.477 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:51:56.478 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2023-11-16T01:51:56.489 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2023-11-16T01:51:55.298728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786984,"kb_used_data":4408,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564444056,"statfs":{"total":579820584960,"available":577990713344,"internally_reserved":0,"allocated":4513792,"data_stored":1938348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.655978+0000","last_change":"2023-11-16T01:48:11.314302+0000","last_active":"2023-11-16T01:50:51.655978+0000","last_peered":"2023-11-16T01:50:51.655978+0000","last_clean":"2023-11-16T01:50:51.655978+0000","last_became_active":"2023-11-16T01:48:11.313884+0000","last_became_peered":"2023-11-16T01:48:11.313884+0000","last_unstale":"2023-11-16T01:50:51.655978+0000","last_undegraded":"2023-11-16T01:50:51.655978+0000","last_fullsized":"2023-11-16T01:50:51.655978+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:10.292390+0000","last_clean_scrub_stamp":"2023-11-16T01:48:10.292390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.597091+0000","last_change":"2023-11-16T01:48:12.316173+0000","last_active":"2023-11-16T01:50:51.597091+0000","last_peered":"2023-11-16T01:50:51.597091+0000","last_clean":"2023-11-16T01:50:51.597091+0000","last_became_active":"2023-11-16T01:48:12.315578+0000","last_became_peered":"2023-11-16T01:48:12.315578+0000","last_unstale":"2023-11-16T01:50:51.597091+0000","last_undegraded":"2023-11-16T01:50:51.597091+0000","last_fullsized":"2023-11-16T01:50:51.597091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.655825+0000","last_change":"2023-11-16T01:48:12.314951+0000","last_active":"2023-11-16T01:50:51.655825+0000","last_peered":"2023-11-16T01:50:51.655825+0000","last_clean":"2023-11-16T01:50:51.655825+0000","last_became_active":"2023-11-16T01:48:12.314325+0000","last_became_peered":"2023-11-16T01:48:12.314325+0000","last_unstale":"2023-11-16T01:50:51.655825+0000","last_undegraded":"2023-11-16T01:50:51.655825+0000","last_fullsized":"2023-11-16T01:50:51.655825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":51,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.758691+0000","last_change":"2023-11-16T01:48:12.314855+0000","last_active":"2023-11-16T01:50:51.758691+0000","last_peered":"2023-11-16T01:50:51.758691+0000","last_clean":"2023-11-16T01:50:51.758691+0000","last_became_active":"2023-11-16T01:48:12.314447+0000","last_became_peered":"2023-11-16T01:48:12.314447+0000","last_unstale":"2023-11-16T01:50:51.758691+0000","last_undegraded":"2023-11-16T01:50:51.758691+0000","last_fullsized":"2023-11-16T01:50:51.758691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":56,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.597094+0000","last_change":"2023-11-16T01:48:12.315525+0000","last_active":"2023-11-16T01:50:51.597094+0000","last_peered":"2023-11-16T01:50:51.597094+0000","last_clean":"2023-11-16T01:50:51.597094+0000","last_became_active":"2023-11-16T01:48:12.314786+0000","last_became_peered":"2023-11-16T01:48:12.314786+0000","last_unstale":"2023-11-16T01:50:51.597094+0000","last_undegraded":"2023-11-16T01:50:51.597094+0000","last_fullsized":"2023-11-16T01:50:51.597094+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.598265+0000","last_change":"2023-11-16T01:48:12.740510+0000","last_active":"2023-11-16T01:50:51.598265+0000","last_peered":"2023-11-16T01:50:51.598265+0000","last_clean":"2023-11-16T01:50:51.598265+0000","last_became_active":"2023-11-16T01:48:12.739242+0000","last_became_peered":"2023-11-16T01:48:12.739242+0000","last_unstale":"2023-11-16T01:50:51.598265+0000","last_undegraded":"2023-11-16T01:50:51.598265+0000","last_fullsized":"2023-11-16T01:50:51.598265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.655983+0000","last_change":"2023-11-16T01:48:12.740239+0000","last_active":"2023-11-16T01:50:51.655983+0000","last_peered":"2023-11-16T01:50:51.655983+0000","last_clean":"2023-11-16T01:50:51.655983+0000","last_became_active":"2023-11-16T01:48:12.739424+0000","last_became_peered":"2023-11-16T01:48:12.739424+0000","last_unstale":"2023-11-16T01:50:51.655983+0000","last_undegraded":"2023-11-16T01:50:51.655983+0000","last_fullsized":"2023-11-16T01:50:51.655983+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.598353+0000","last_change":"2023-11-16T01:48:12.315863+0000","last_active":"2023-11-16T01:50:51.598353+0000","last_peered":"2023-11-16T01:50:51.598353+0000","last_clean":"2023-11-16T01:50:51.598353+0000","last_became_active":"2023-11-16T01:48:12.315453+0000","last_became_peered":"2023-11-16T01:48:12.315453+0000","last_unstale":"2023-11-16T01:50:51.598353+0000","last_undegraded":"2023-11-16T01:50:51.598353+0000","last_fullsized":"2023-11-16T01:50:51.598353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":53,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-16T01:50:51.656031+0000","last_change":"2023-11-16T01:48:12.945654+0000","last_active":"2023-11-16T01:50:51.656031+0000","last_peered":"2023-11-16T01:50:51.656031+0000","last_clean":"2023-11-16T01:50:51.656031+0000","last_became_active":"2023-11-16T01:48:12.944985+0000","last_became_peered":"2023-11-16T01:48:12.944985+0000","last_unstale":"2023-11-16T01:50:51.656031+0000","last_undegraded":"2023-11-16T01:50:51.656031+0000","last_fullsized":"2023-11-16T01:50:51.656031+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:48:11.295280+0000","last_clean_scrub_stamp":"2023-11-16T01:48:11.295280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297828,"kb_used_data":732,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074012,"statfs":{"total":96636764160,"available":96331788288,"internally_reserved":0,"allocated":749568,"data_stored":322412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73499999999999999,"15min":0.71899999999999997},"min":{"1min":0.57199999999999995,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.95899999999999996,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.75600000000000001,"15min":0.73499999999999999},"min":{"1min":0.66300000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.65600000000000003}]},{"osd":1,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.71999999999999997,"15min":0.70399999999999996},"min":{"1min":0.56599999999999995,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.89700000000000002,"5min":1.085,"15min":1.085},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.77000000000000002,"15min":0.76200000000000001},"min":{"1min":0.71299999999999997,"5min":0.317,"15min":0.317},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.86399999999999999}]},{"osd":2,"last update":"Thu Nov 16 01:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.755,"15min":0.72699999999999998},"min":{"1min":0.60199999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0149999999999999,"5min":1.369,"15min":1.369},"last":1.002},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.82799999999999996,"15min":0.80700000000000005},"min":{"1min":0.74199999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97699999999999998,"5min":1.302,"15min":1.302},"last":0.92800000000000005}]},{"osd":3,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63600000000000001,"15min":0.59999999999999998},"min":{"1min":0.47199999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.96999999999999997,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.69099999999999995,"15min":0.67600000000000005},"min":{"1min":0.53100000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0629999999999999,"5min":2.339,"15min":2.339},"last":0.80300000000000005}]},{"osd":4,"last update":"Thu Nov 16 01:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.747,"15min":0.748},"min":{"1min":0.54200000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.873,"5min":1.056,"15min":1.056},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.80600000000000005,"15min":0.79400000000000004},"min":{"1min":0.63,"5min":0.437,"15min":0.437},"max":{"1min":1.1120000000000001,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.97699999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640302,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.69999999999999996,"15min":0.67400000000000004},"min":{"1min":0.53000000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71099999999999997,"15min":0.69799999999999995},"min":{"1min":0.56599999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.0249999999999999,"5min":1.052,"15min":1.052},"last":0.77000000000000002}]},{"osd":1,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79400000000000004,"15min":0.78300000000000003},"min":{"1min":0.66300000000000003,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.96899999999999997,"5min":1.145,"15min":1.145},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.83999999999999997,"15min":0.82599999999999996},"min":{"1min":0.69999999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.024,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":1.0009999999999999}]},{"osd":2,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.75900000000000001,"15min":0.74099999999999999},"min":{"1min":0.628,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.97699999999999998,"5min":1.163,"15min":1.163},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.751,"15min":0.74099999999999999},"min":{"1min":0.60399999999999998,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.94299999999999995,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.75600000000000001}]},{"osd":3,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.66200000000000003,"15min":0.66000000000000003},"min":{"1min":0.434,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89700000000000002,"5min":1.087,"15min":1.087},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.72599999999999998,"15min":0.71999999999999997},"min":{"1min":0.56699999999999995,"5min":0.313,"15min":0.313},"max":{"1min":0.85499999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.78700000000000003}]},{"osd":5,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73099999999999998,"15min":0.72399999999999998},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0069999999999999,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.80200000000000005,"15min":0.78800000000000003},"min":{"1min":0.60199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.2769999999999999,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":1.0429999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640302,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70999999999999996,"15min":0.68300000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90300000000000002,"5min":1.014,"15min":1.014},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.75800000000000001,"15min":0.72999999999999998},"min":{"1min":0.52500000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.86299999999999999}]},{"osd":1,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.749,"15min":0.73099999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0289999999999999,"5min":1.129,"15min":1.129},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.755,"5min":0.75800000000000001,"15min":0.74199999999999999},"min":{"1min":0.59399999999999997,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.97699999999999998,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.83399999999999996}]},{"osd":2,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.745,"15min":0.72099999999999997},"min":{"1min":0.59499999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.093,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.81499999999999995,"15min":0.78400000000000003},"min":{"1min":0.60099999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.90600000000000003}]},{"osd":4,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65200000000000002,"15min":0.64000000000000001},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89600000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.54700000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.98399999999999999,"5min":1.218,"15min":1.218},"last":0.63}]},{"osd":5,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.78400000000000003,"15min":0.745},"min":{"1min":0.56699999999999995,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.85099999999999998,"15min":0.83599999999999997},"min":{"1min":0.70699999999999996,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.079,"5min":1.409,"15min":1.409},"last":0.73199999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.47199999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.65500000000000003,"5min":0.747,"15min":0.747},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57499999999999996,"15min":0.56200000000000006},"min":{"1min":0.48299999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.61399999999999999}]},{"osd":1,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.68500000000000005,"15min":0.69399999999999995},"min":{"1min":0.44600000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.92800000000000005,"5min":0.997,"15min":0.997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.69699999999999995,"15min":0.69299999999999995},"min":{"1min":0.503,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.67200000000000004}]},{"osd":3,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.78500000000000003,"15min":0.73199999999999998},"min":{"1min":0.63900000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.1419999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86499999999999999,"15min":0.84299999999999997},"min":{"1min":0.71399999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.016,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.90900000000000003}]},{"osd":4,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84899999999999998,"15min":0.83299999999999996},"min":{"1min":0.626,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.3280000000000001,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.82699999999999996,"15min":0.80300000000000005},"min":{"1min":0.64100000000000001,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.3,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.80900000000000005}]},{"osd":5,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.78400000000000003,"15min":0.754},"min":{"1min":0.67400000000000004,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.84399999999999997,"15min":0.81599999999999995},"min":{"1min":0.70799999999999996,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.76100000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640303,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297828,"kb_used_data":732,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074012,"statfs":{"total":96636764160,"available":96331788288,"internally_reserved":0,"allocated":749568,"data_stored":322412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71599999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.64700000000000002,"15min":0.63500000000000001},"min":{"1min":0.50900000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.70599999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.67800000000000005}]},{"osd":3,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.80200000000000005,"15min":0.77600000000000002},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96399999999999997,"5min":1.127,"15min":1.127},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.879,"5min":0.83099999999999996,"15min":0.80900000000000005},"min":{"1min":0.76700000000000002,"5min":0.62,"15min":0.62},"max":{"1min":1.1379999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.79100000000000004}]},{"osd":4,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.80800000000000005,"15min":0.80200000000000005},"min":{"1min":0.65600000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.98199999999999998,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.84199999999999997,"15min":0.83199999999999996},"min":{"1min":0.67300000000000004,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.84899999999999998}]},{"osd":5,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.82599999999999996,"15min":0.81399999999999995},"min":{"1min":0.69799999999999995,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.85799999999999998,"15min":0.84299999999999997},"min":{"1min":0.73999999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.1080000000000001,"5min":1.403,"15min":1.403},"last":0.77300000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640305,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.47599999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.90200000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.47899999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.48599999999999999}]},{"osd":2,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.52700000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.93500000000000005,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.31},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.44900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.95499999999999996,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.45400000000000001}]},{"osd":3,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.81100000000000005,"15min":0.78100000000000003},"min":{"1min":0.70699999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.84699999999999998,"15min":0.82699999999999996},"min":{"1min":0.71499999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.76500000000000001}]},{"osd":4,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79000000000000004},"min":{"1min":0.61299999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0349999999999999,"5min":1.242,"15min":1.242},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.79800000000000004,"15min":0.76400000000000001},"min":{"1min":0.73999999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.65300000000000002}]},{"osd":5,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.85199999999999998,"15min":0.83799999999999997},"min":{"1min":0.71399999999999997,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1419999999999999,"5min":1.147,"15min":1.147},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.86799999999999999,"15min":0.85199999999999998},"min":{"1min":0.71599999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.032,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.58299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T01:51:56.492 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-11-16T01:48:10.292390+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=10, tm_wday=3, tm_yday=320, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=51, tm_sec=52, tm_wday=3, tm_yday=320, tm_isdst=0) 2023-11-16T01:51:56.493 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-11-16T01:48:11.295280+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=11, tm_wday=3, tm_yday=320, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=51, tm_sec=52, tm_wday=3, tm_yday=320, tm_isdst=0) 2023-11-16T01:51:56.493 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-11-16T01:48:11.295280+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=11, tm_wday=3, tm_yday=320, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=51, tm_sec=52, tm_wday=3, tm_yday=320, tm_isdst=0) 2023-11-16T01:51:56.494 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-11-16T01:48:11.295280+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=11, tm_wday=3, tm_yday=320, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=51, tm_sec=52, tm_wday=3, tm_yday=320, tm_isdst=0) 2023-11-16T01:51:56.494 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-11-16T01:48:11.295280+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=11, tm_wday=3, tm_yday=320, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=51, tm_sec=52, tm_wday=3, tm_yday=320, tm_isdst=0) 2023-11-16T01:51:56.494 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-11-16T01:48:11.295280+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=11, tm_wday=3, tm_yday=320, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=51, tm_sec=52, tm_wday=3, tm_yday=320, tm_isdst=0) 2023-11-16T01:51:56.494 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-11-16T01:48:11.295280+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=11, tm_wday=3, tm_yday=320, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=51, tm_sec=52, tm_wday=3, tm_yday=320, tm_isdst=0) 2023-11-16T01:51:56.495 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-11-16T01:48:11.295280+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=11, tm_wday=3, tm_yday=320, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=51, tm_sec=52, tm_wday=3, tm_yday=320, tm_isdst=0) 2023-11-16T01:51:56.495 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-11-16T01:48:11.295280+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=11, tm_wday=3, tm_yday=320, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=16, tm_hour=1, tm_min=51, tm_sec=52, tm_wday=3, tm_yday=320, tm_isdst=0) 2023-11-16T01:51:56.495 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-11-16T01:52:16.496 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T01:52:16.783 INFO:teuthology.orchestra.run.smithi083.stdout: 2023-11-16T01:52:16.786 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2023-11-16T01:52:16.797 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2023-11-16T01:52:15.303885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786984,"kb_used_data":4408,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564444056,"statfs":{"total":579820584960,"available":577990713344,"internally_reserved":0,"allocated":4513792,"data_stored":1938348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-16T01:51:57.373787+0000","last_change":"2023-11-16T01:51:57.373787+0000","last_active":"2023-11-16T01:51:57.373787+0000","last_peered":"2023-11-16T01:51:57.373787+0000","last_clean":"2023-11-16T01:51:57.373787+0000","last_became_active":"2023-11-16T01:48:11.313884+0000","last_became_peered":"2023-11-16T01:48:11.313884+0000","last_unstale":"2023-11-16T01:51:57.373787+0000","last_undegraded":"2023-11-16T01:51:57.373787+0000","last_fullsized":"2023-11-16T01:51:57.373787+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:51:57.373573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:51:57.373573+0000","last_clean_scrub_stamp":"2023-11-16T01:51:57.373573+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":61,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-16T01:52:01.350946+0000","last_change":"2023-11-16T01:52:01.350946+0000","last_active":"2023-11-16T01:52:01.350946+0000","last_peered":"2023-11-16T01:52:01.350946+0000","last_clean":"2023-11-16T01:52:01.350946+0000","last_became_active":"2023-11-16T01:48:12.315578+0000","last_became_peered":"2023-11-16T01:48:12.315578+0000","last_unstale":"2023-11-16T01:52:01.350946+0000","last_undegraded":"2023-11-16T01:52:01.350946+0000","last_fullsized":"2023-11-16T01:52:01.350946+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:52:01.350672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:52:01.350672+0000","last_clean_scrub_stamp":"2023-11-16T01:52:01.350672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":58,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-16T01:52:06.327039+0000","last_change":"2023-11-16T01:52:06.327039+0000","last_active":"2023-11-16T01:52:06.327039+0000","last_peered":"2023-11-16T01:52:06.327039+0000","last_clean":"2023-11-16T01:52:06.327039+0000","last_became_active":"2023-11-16T01:48:12.314325+0000","last_became_peered":"2023-11-16T01:48:12.314325+0000","last_unstale":"2023-11-16T01:52:06.327039+0000","last_undegraded":"2023-11-16T01:52:06.327039+0000","last_fullsized":"2023-11-16T01:52:06.327039+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:52:06.326805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:52:06.326805+0000","last_clean_scrub_stamp":"2023-11-16T01:52:06.326805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":56,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-16T01:52:02.128597+0000","last_change":"2023-11-16T01:52:02.128597+0000","last_active":"2023-11-16T01:52:02.128597+0000","last_peered":"2023-11-16T01:52:02.128597+0000","last_clean":"2023-11-16T01:52:02.128597+0000","last_became_active":"2023-11-16T01:48:12.314447+0000","last_became_peered":"2023-11-16T01:48:12.314447+0000","last_unstale":"2023-11-16T01:52:02.128597+0000","last_undegraded":"2023-11-16T01:52:02.128597+0000","last_fullsized":"2023-11-16T01:52:02.128597+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:52:02.128354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:52:02.128354+0000","last_clean_scrub_stamp":"2023-11-16T01:52:02.128354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":61,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-16T01:52:00.378224+0000","last_change":"2023-11-16T01:52:00.378224+0000","last_active":"2023-11-16T01:52:00.378224+0000","last_peered":"2023-11-16T01:52:00.378224+0000","last_clean":"2023-11-16T01:52:00.378224+0000","last_became_active":"2023-11-16T01:48:12.314786+0000","last_became_peered":"2023-11-16T01:48:12.314786+0000","last_unstale":"2023-11-16T01:52:00.378224+0000","last_undegraded":"2023-11-16T01:52:00.378224+0000","last_fullsized":"2023-11-16T01:52:00.378224+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:52:00.377870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:52:00.377870+0000","last_clean_scrub_stamp":"2023-11-16T01:52:00.377870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":57,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-16T01:51:55.939508+0000","last_change":"2023-11-16T01:51:55.939508+0000","last_active":"2023-11-16T01:51:55.939508+0000","last_peered":"2023-11-16T01:51:55.939508+0000","last_clean":"2023-11-16T01:51:55.939508+0000","last_became_active":"2023-11-16T01:48:12.739242+0000","last_became_peered":"2023-11-16T01:48:12.739242+0000","last_unstale":"2023-11-16T01:51:55.939508+0000","last_undegraded":"2023-11-16T01:51:55.939508+0000","last_fullsized":"2023-11-16T01:51:55.939508+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:51:55.939289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:51:55.939289+0000","last_clean_scrub_stamp":"2023-11-16T01:51:55.939289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":58,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-16T01:52:09.321043+0000","last_change":"2023-11-16T01:52:09.321043+0000","last_active":"2023-11-16T01:52:09.321043+0000","last_peered":"2023-11-16T01:52:09.321043+0000","last_clean":"2023-11-16T01:52:09.321043+0000","last_became_active":"2023-11-16T01:48:12.739424+0000","last_became_peered":"2023-11-16T01:48:12.739424+0000","last_unstale":"2023-11-16T01:52:09.321043+0000","last_undegraded":"2023-11-16T01:52:09.321043+0000","last_fullsized":"2023-11-16T01:52:09.321043+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:52:09.320822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:52:09.320822+0000","last_clean_scrub_stamp":"2023-11-16T01:52:09.320822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":57,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-16T01:51:58.923125+0000","last_change":"2023-11-16T01:51:58.923125+0000","last_active":"2023-11-16T01:51:58.923125+0000","last_peered":"2023-11-16T01:51:58.923125+0000","last_clean":"2023-11-16T01:51:58.923125+0000","last_became_active":"2023-11-16T01:48:12.315453+0000","last_became_peered":"2023-11-16T01:48:12.315453+0000","last_unstale":"2023-11-16T01:51:58.923125+0000","last_undegraded":"2023-11-16T01:51:58.923125+0000","last_fullsized":"2023-11-16T01:51:58.923125+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:51:58.922880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:51:58.922880+0000","last_clean_scrub_stamp":"2023-11-16T01:51:58.922880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":58,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-16T01:52:10.282521+0000","last_change":"2023-11-16T01:52:10.282521+0000","last_active":"2023-11-16T01:52:10.282521+0000","last_peered":"2023-11-16T01:52:10.282521+0000","last_clean":"2023-11-16T01:52:10.282521+0000","last_became_active":"2023-11-16T01:48:12.944985+0000","last_became_peered":"2023-11-16T01:48:12.944985+0000","last_unstale":"2023-11-16T01:52:10.282521+0000","last_undegraded":"2023-11-16T01:52:10.282521+0000","last_fullsized":"2023-11-16T01:52:10.282521+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T01:52:10.282300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T01:52:10.282300+0000","last_clean_scrub_stamp":"2023-11-16T01:52:10.282300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640309,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297828,"kb_used_data":732,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074012,"statfs":{"total":96636764160,"available":96331788288,"internally_reserved":0,"allocated":749568,"data_stored":322412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73499999999999999,"15min":0.71899999999999997},"min":{"1min":0.57199999999999995,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.95899999999999996,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.75600000000000001,"15min":0.73499999999999999},"min":{"1min":0.66300000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.70299999999999996}]},{"osd":1,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.71999999999999997,"15min":0.70399999999999996},"min":{"1min":0.56599999999999995,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.89700000000000002,"5min":1.085,"15min":1.085},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.77000000000000002,"15min":0.76200000000000001},"min":{"1min":0.71299999999999997,"5min":0.317,"15min":0.317},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.80700000000000005}]},{"osd":2,"last update":"Thu Nov 16 01:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.755,"15min":0.72699999999999998},"min":{"1min":0.60199999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0149999999999999,"5min":1.369,"15min":1.369},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.82799999999999996,"15min":0.80700000000000005},"min":{"1min":0.74199999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97699999999999998,"5min":1.302,"15min":1.302},"last":0.73399999999999999}]},{"osd":3,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63600000000000001,"15min":0.59999999999999998},"min":{"1min":0.47199999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.96999999999999997,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.69099999999999995,"15min":0.67600000000000005},"min":{"1min":0.53100000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0629999999999999,"5min":2.339,"15min":2.339},"last":0.82499999999999996}]},{"osd":4,"last update":"Thu Nov 16 01:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.747,"15min":0.748},"min":{"1min":0.54200000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.873,"5min":1.056,"15min":1.056},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.80600000000000005,"15min":0.79400000000000004},"min":{"1min":0.63,"5min":0.437,"15min":0.437},"max":{"1min":1.1120000000000001,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.90900000000000003}]}]},{"osd":4,"up_from":11,"seq":47244640306,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.69999999999999996,"15min":0.67400000000000004},"min":{"1min":0.53000000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71099999999999997,"15min":0.69799999999999995},"min":{"1min":0.56599999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.0249999999999999,"5min":1.052,"15min":1.052},"last":0.745}]},{"osd":1,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79400000000000004,"15min":0.78300000000000003},"min":{"1min":0.66300000000000003,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.96899999999999997,"5min":1.145,"15min":1.145},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.83999999999999997,"15min":0.82599999999999996},"min":{"1min":0.69999999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.024,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.89800000000000002}]},{"osd":2,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.75900000000000001,"15min":0.74099999999999999},"min":{"1min":0.628,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.97699999999999998,"5min":1.163,"15min":1.163},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.751,"15min":0.74099999999999999},"min":{"1min":0.60399999999999998,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.94299999999999995,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.79400000000000004}]},{"osd":3,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.66200000000000003,"15min":0.66000000000000003},"min":{"1min":0.434,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89700000000000002,"5min":1.087,"15min":1.087},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.72599999999999998,"15min":0.71999999999999997},"min":{"1min":0.56699999999999995,"5min":0.313,"15min":0.313},"max":{"1min":0.85499999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.67400000000000004}]},{"osd":5,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73099999999999998,"15min":0.72399999999999998},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0069999999999999,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.80200000000000005,"15min":0.78800000000000003},"min":{"1min":0.60199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.2769999999999999,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.76900000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640306,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70999999999999996,"15min":0.68300000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90300000000000002,"5min":1.014,"15min":1.014},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.75800000000000001,"15min":0.72999999999999998},"min":{"1min":0.52500000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.70099999999999996}]},{"osd":1,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.749,"15min":0.73099999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0289999999999999,"5min":1.129,"15min":1.129},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.755,"5min":0.75800000000000001,"15min":0.74199999999999999},"min":{"1min":0.59399999999999997,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.97699999999999998,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.871}]},{"osd":2,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.745,"15min":0.72099999999999997},"min":{"1min":0.59499999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.093,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.81499999999999995,"15min":0.78400000000000003},"min":{"1min":0.60099999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.89600000000000002}]},{"osd":4,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65200000000000002,"15min":0.64000000000000001},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89600000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.754},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.54700000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.98399999999999999,"5min":1.218,"15min":1.218},"last":0.73099999999999998}]},{"osd":5,"last update":"Thu Nov 16 01:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.78400000000000003,"15min":0.745},"min":{"1min":0.56699999999999995,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.85099999999999998,"15min":0.83599999999999997},"min":{"1min":0.70699999999999996,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.079,"5min":1.409,"15min":1.409},"last":0.93300000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640307,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.47199999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.65500000000000003,"5min":0.747,"15min":0.747},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57499999999999996,"15min":0.56200000000000006},"min":{"1min":0.48299999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.621}]},{"osd":1,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.68500000000000005,"15min":0.69399999999999995},"min":{"1min":0.44600000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.92800000000000005,"5min":0.997,"15min":0.997},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.69699999999999995,"15min":0.69299999999999995},"min":{"1min":0.503,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.75900000000000001}]},{"osd":3,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.78500000000000003,"15min":0.73199999999999998},"min":{"1min":0.63900000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.1419999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86499999999999999,"15min":0.84299999999999997},"min":{"1min":0.71399999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.016,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.83599999999999997}]},{"osd":4,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84899999999999998,"15min":0.83299999999999996},"min":{"1min":0.626,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.3280000000000001,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.82699999999999996,"15min":0.80300000000000005},"min":{"1min":0.64100000000000001,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.3,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.86799999999999999}]},{"osd":5,"last update":"Thu Nov 16 01:51:12 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.78400000000000003,"15min":0.754},"min":{"1min":0.67400000000000004,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.84399999999999997,"15min":0.81599999999999995},"min":{"1min":0.70799999999999996,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.94299999999999995}]}]},{"osd":1,"up_from":11,"seq":47244640307,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297828,"kb_used_data":732,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074012,"statfs":{"total":96636764160,"available":96331788288,"internally_reserved":0,"allocated":749568,"data_stored":322412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71599999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.51700000000000002}]},{"osd":2,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.64700000000000002,"15min":0.63500000000000001},"min":{"1min":0.50900000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.502},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.70599999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.72399999999999998}]},{"osd":3,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.80200000000000005,"15min":0.77600000000000002},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96399999999999997,"5min":1.127,"15min":1.127},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.879,"5min":0.83099999999999996,"15min":0.80900000000000005},"min":{"1min":0.76700000000000002,"5min":0.62,"15min":0.62},"max":{"1min":1.1379999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.85799999999999998}]},{"osd":4,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.80800000000000005,"15min":0.80200000000000005},"min":{"1min":0.65600000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.98199999999999998,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.75},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.84199999999999997,"15min":0.83199999999999996},"min":{"1min":0.67300000000000004,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.80000000000000004}]},{"osd":5,"last update":"Thu Nov 16 01:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.82599999999999996,"15min":0.81399999999999995},"min":{"1min":0.69799999999999995,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.85799999999999998,"15min":0.84299999999999997},"min":{"1min":0.73999999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.1080000000000001,"5min":1.403,"15min":1.403},"last":0.67400000000000004}]}]},{"osd":0,"up_from":11,"seq":47244640309,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297832,"kb_used_data":736,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074008,"statfs":{"total":96636764160,"available":96331784192,"internally_reserved":0,"allocated":753664,"data_stored":323381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.47599999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.90200000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.47899999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.52700000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.93500000000000005,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.44900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.95499999999999996,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.67800000000000005}]},{"osd":3,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.81100000000000005,"15min":0.78100000000000003},"min":{"1min":0.70699999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.876},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.84699999999999998,"15min":0.82699999999999996},"min":{"1min":0.71499999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.65700000000000003}]},{"osd":4,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79000000000000004},"min":{"1min":0.61299999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0349999999999999,"5min":1.242,"15min":1.242},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.79800000000000004,"15min":0.76400000000000001},"min":{"1min":0.73999999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.73099999999999998}]},{"osd":5,"last update":"Thu Nov 16 01:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.85199999999999998,"15min":0.83799999999999997},"min":{"1min":0.71399999999999997,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1419999999999999,"5min":1.147,"15min":1.147},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.86799999999999999,"15min":0.85199999999999998},"min":{"1min":0.71599999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.032,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":1.0249999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T01:52:16.800 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-11-16T01:52:17.112 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-16T01:52:17.112 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-16T01:52:17.113 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-16T01:52:17.113 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T01:52:17.190 INFO:tasks.ceph.osd.0:Stopped 2023-11-16T01:52:17.190 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-16T01:52:17.190 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T01:52:17.277 INFO:tasks.ceph.osd.1:Stopped 2023-11-16T01:52:17.277 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-16T01:52:17.278 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T01:52:17.352 INFO:tasks.ceph.osd.2:Stopped 2023-11-16T01:52:17.353 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-11-16T01:52:17.353 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T01:52:17.426 INFO:tasks.ceph.osd.3:Stopped 2023-11-16T01:52:17.426 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-11-16T01:52:17.426 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T01:52:17.527 INFO:tasks.ceph.osd.4:Stopped 2023-11-16T01:52:17.529 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-11-16T01:52:17.529 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T01:52:17.602 INFO:tasks.ceph.osd.5:Stopped 2023-11-16T01:52:17.603 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-16T01:52:17.603 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-16T01:52:17.603 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T01:52:17.633 INFO:tasks.ceph.mgr.x:Stopped 2023-11-16T01:52:17.633 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-16T01:52:17.634 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-16T01:52:17.634 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T01:52:17.655 INFO:tasks.ceph.mon.a:Stopped 2023-11-16T01:52:17.655 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-16T01:52:17.655 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T01:52:17.671 INFO:tasks.ceph.mon.c:Stopped 2023-11-16T01:52:17.671 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-16T01:52:17.671 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T01:52:17.694 INFO:tasks.ceph.mon.b:Stopped 2023-11-16T01:52:17.694 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-16T01:52:17.695 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2023-11-16T01:52:17.716 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi083.front.sepia.ceph.com 2023-11-16T01:52:17.717 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-16T01:52:18.118 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi083.front.sepia.ceph.com 2023-11-16T01:52:18.118 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-16T01:52:18.393 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi083.front.sepia.ceph.com 2023-11-16T01:52:18.393 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-16T01:52:18.560 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi136.front.sepia.ceph.com 2023-11-16T01:52:18.560 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-11-16T01:52:18.775 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi136.front.sepia.ceph.com 2023-11-16T01:52:18.775 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-11-16T01:52:19.037 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi136.front.sepia.ceph.com 2023-11-16T01:52:19.037 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-11-16T01:52:19.253 INFO:tasks.ceph:Archiving mon data... 2023-11-16T01:52:19.254 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694/data/mon.a.tgz 2023-11-16T01:52:19.255 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2023-11-16T01:52:19.261 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.TO7AlN5sYM 2023-11-16T01:52:19.262 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.TO7AlN5sYM 2023-11-16T01:52:19.429 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.TO7AlN5sYM 2023-11-16T01:52:19.450 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.TO7AlN5sYM is 506KB 2023-11-16T01:52:19.512 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.TO7AlN5sYM 2023-11-16T01:52:19.518 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694/data/mon.c.tgz 2023-11-16T01:52:19.518 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2023-11-16T01:52:19.525 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.SbpVn49Rec 2023-11-16T01:52:19.526 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.SbpVn49Rec 2023-11-16T01:52:19.653 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.SbpVn49Rec 2023-11-16T01:52:19.674 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.SbpVn49Rec is 518KB 2023-11-16T01:52:19.736 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.SbpVn49Rec 2023-11-16T01:52:19.742 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694/data/mon.b.tgz 2023-11-16T01:52:19.743 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2023-11-16T01:52:19.749 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.8WKmkOAVv3 2023-11-16T01:52:19.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.8WKmkOAVv3 2023-11-16T01:52:19.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.8WKmkOAVv3 2023-11-16T01:52:19.942 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.8WKmkOAVv3 is 517KB 2023-11-16T01:52:20.010 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.8WKmkOAVv3 2023-11-16T01:52:20.016 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-16T01:52:20.016 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-16T01:52:20.019 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-16T01:52:20.076 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-16T01:52:20.077 DEBUG:teuthology.orchestra.run.smithi083:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-16T01:52:20.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-16T01:52:20.103 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-16T01:52:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-16T01:52:20.116 INFO:tasks.ceph:Archiving crash dumps... 2023-11-16T01:52:20.119 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694/remote/smithi083/crash 2023-11-16T01:52:20.120 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-16T01:52:20.135 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694/remote/smithi136/crash 2023-11-16T01:52:20.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-16T01:52:20.172 INFO:tasks.ceph:Compressing logs... 2023-11-16T01:52:20.172 DEBUG:teuthology.orchestra.run.smithi083:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-16T01:52:20.175 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-16T01:52:20.201 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34612.log 2023-11-16T01:52:20.201 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34269.log 2023-11-16T01:52:20.203 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34612.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.34612.log.gz 2023-11-16T01:52:20.203 INFO:teuthology.orchestra.run.smithi136.stderr:--verbose -- /var/log/ceph/ceph-client.admin.34915.log 2023-11-16T01:52:20.203 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33883.log 2023-11-16T01:52:20.204 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34269.log.gz 2023-11-16T01:52:20.204 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34026.log 2023-11-16T01:52:20.204 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34915.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-11-16T01:52:20.204 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34915.log.gz 2023-11-16T01:52:20.204 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33883.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33883.log.gz 2023-11-16T01:52:20.209 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36125.log 2023-11-16T01:52:20.209 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34181.log 2023-11-16T01:52:20.216 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36125.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.36125.log.gz-- 2023-11-16T01:52:20.216 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35963.log 2023-11-16T01:52:20.216 INFO:teuthology.orchestra.run.smithi083.stderr:gzip /var/log/ceph/ceph-client.admin.34181.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35204.log 2023-11-16T01:52:20.216 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34181.log.gz 2023-11-16T01:52:20.217 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33452.log 2023-11-16T01:52:20.217 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35963.log.gz 2023-11-16T01:52:20.217 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35801.log /var/log/ceph/ceph-client.admin.35204.log: 2023-11-16T01:52:20.217 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35204.log.gz 2023-11-16T01:52:20.217 INFO:teuthology.orchestra.run.smithi083.stderr:gzip /var/log/ceph/ceph-client.admin.33452.log: -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-16T01:52:20.217 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33452.log.gz 2023-11-16T01:52:20.218 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33965.log 2023-11-16T01:52:20.218 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35801.log.gz 2023-11-16T01:52:20.218 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35231.log 2023-11-16T01:52:20.218 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.33965.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.33965.log.gz 2023-11-16T01:52:20.218 INFO:teuthology.orchestra.run.smithi083.stderr:--verbose -- /var/log/ceph/ceph-client.admin.34289.log 2023-11-16T01:52:20.218 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34235.log 2023-11-16T01:52:20.271 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34026.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34026.log.gz 2023-11-16T01:52:20.271 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35392.log 2023-11-16T01:52:20.271 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30896.log 2023-11-16T01:52:20.271 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.35392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34659.log 2023-11-16T01:52:20.271 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35392.log.gz 2023-11-16T01:52:20.271 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34076.log 2023-11-16T01:52:20.272 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.30896.log: /var/log/ceph/ceph-client.admin.34659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30896.log.gz 2023-11-16T01:52:20.272 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34659.log.gz 2023-11-16T01:52:20.272 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35090.log 2023-11-16T01:52:20.272 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34184.log 2023-11-16T01:52:20.272 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35090.log: /var/log/ceph/ceph-client.admin.34076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35090.log.gz 2023-11-16T01:52:20.272 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34076.log.gz 2023-11-16T01:52:20.272 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33714.log 2023-11-16T01:52:20.272 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34184.log.gz 2023-11-16T01:52:20.273 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35194.log 2023-11-16T01:52:20.273 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33714.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33714.log.gz 2023-11-16T01:52:20.273 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33931.log 2023-11-16T01:52:20.273 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34840.log 2023-11-16T01:52:20.273 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35194.log.gz 2023-11-16T01:52:20.273 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34568.log 2023-11-16T01:52:20.273 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33931.log: /var/log/ceph/ceph-client.admin.34840.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.33931.log.gz 0.0%-- 2023-11-16T01:52:20.273 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34840.log.gz 2023-11-16T01:52:20.274 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34865.log 2023-11-16T01:52:20.274 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34568.log.gz 2023-11-16T01:52:20.274 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35040.log 2023-11-16T01:52:20.274 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34865.log.gz 2023-11-16T01:52:20.274 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-11-16T01:52:20.274 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34990.log 2023-11-16T01:52:20.274 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34428.log 2023-11-16T01:52:20.274 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.5.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35040.log.gz 2023-11-16T01:52:20.275 INFO:teuthology.orchestra.run.smithi136.stderr:gzip /var/log/ceph/ceph-client.admin.34990.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34634.log 2023-11-16T01:52:20.275 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34990.log.gz 2023-11-16T01:52:20.275 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34428.log.gz 2023-11-16T01:52:20.276 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-16T01:52:20.276 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34590.log 2023-11-16T01:52:20.276 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34634.log.gz 2023-11-16T01:52:20.276 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30870.log 2023-11-16T01:52:20.276 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.34590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35442.log 2023-11-16T01:52:20.276 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34590.log.gz 2023-11-16T01:52:20.276 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-16T01:52:20.276 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.30870.log: /var/log/ceph/ceph-client.admin.35442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30870.log.gz 2023-11-16T01:52:20.277 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35442.log.gz 2023-11-16T01:52:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34153.log 2023-11-16T01:52:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34101.log 2023-11-16T01:52:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33765.log 2023-11-16T01:52:20.277 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34153.log.gz 2023-11-16T01:52:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30781.log 2023-11-16T01:52:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34101.log: /var/log/ceph/ceph-client.admin.33765.log: 89.6% 0.0% -- replaced with /var/log/ceph/ceph.audit.log.gz -- replaced with /var/log/ceph/ceph-client.admin.33765.log.gz 2023-11-16T01:52:20.277 INFO:teuthology.orchestra.run.smithi136.stderr: 2023-11-16T01:52:20.278 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34101.log.gz 2023-11-16T01:52:20.278 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33837.log 2023-11-16T01:52:20.278 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.30781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33956.log 2023-11-16T01:52:20.278 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30781.log.gz 2023-11-16T01:52:20.278 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35316.log 2023-11-16T01:52:20.278 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33837.log.gz 2023-11-16T01:52:20.278 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34890.log 2023-11-16T01:52:20.278 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33956.log.gz 2023-11-16T01:52:20.278 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33860.log 2023-11-16T01:52:20.279 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35316.log: /var/log/ceph/ceph-client.admin.34890.log: 24.9% -- replaced with /var/log/ceph/ceph-client.admin.35316.log.gz 2023-11-16T01:52:20.279 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35221.log 2023-11-16T01:52:20.279 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34890.log.gz 2023-11-16T01:52:20.279 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33860.log: /var/log/ceph/ceph-client.admin.34242.log 2023-11-16T01:52:20.279 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33860.log.gz 2023-11-16T01:52:20.279 INFO:teuthology.orchestra.run.smithi136.stderr:gzip /var/log/ceph/ceph-client.admin.35221.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35417.log 2023-11-16T01:52:20.279 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35221.log.gz 2023-11-16T01:52:20.279 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34346.log 2023-11-16T01:52:20.280 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34242.log.gz 2023-11-16T01:52:20.280 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34684.log 2023-11-16T01:52:20.280 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35417.log.gz 2023-11-16T01:52:20.280 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34051.log 2023-11-16T01:52:20.280 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34346.log.gz 2023-11-16T01:52:20.280 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34759.log 2023-11-16T01:52:20.280 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34734.log 2023-11-16T01:52:20.280 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34684.log.gz 2023-11-16T01:52:20.280 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34051.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34051.log.gz 2023-11-16T01:52:20.281 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33791.log 2023-11-16T01:52:20.281 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34759.log.gz 2023-11-16T01:52:20.281 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34371.log 2023-11-16T01:52:20.281 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34734.log.gz 2023-11-16T01:52:20.281 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33791.log.gz 2023-11-16T01:52:20.281 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34371.log.gz 2023-11-16T01:52:20.281 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34126.log 2023-11-16T01:52:20.281 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34300.log 2023-11-16T01:52:20.282 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34709.log 2023-11-16T01:52:20.282 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34126.log.gz 2023-11-16T01:52:20.282 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.34300.log: -- /var/log/ceph/ceph-client.admin.35367.log 2023-11-16T01:52:20.282 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34300.log.gz 2023-11-16T01:52:20.282 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33740.log 2023-11-16T01:52:20.282 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34709.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34709.log.gz 2023-11-16T01:52:20.282 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30922.log 2023-11-16T01:52:20.282 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35367.log.gz 2023-11-16T01:52:20.283 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33740.log: /var/log/ceph/ceph-client.admin.34965.log 2023-11-16T01:52:20.283 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33740.log.gz 2023-11-16T01:52:20.283 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35141.log 2023-11-16T01:52:20.283 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.30922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30922.log.gz 2023-11-16T01:52:20.283 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35468.log 2023-11-16T01:52:20.283 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34965.log.gz 2023-11-16T01:52:20.283 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33979.log 2023-11-16T01:52:20.283 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35141.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35141.log.gz 2023-11-16T01:52:20.284 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33908.log 2023-11-16T01:52:20.284 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35468.log.gz 2023-11-16T01:52:20.284 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34940.log 2023-11-16T01:52:20.284 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33979.log.gz 2023-11-16T01:52:20.284 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34815.log 2023-11-16T01:52:20.284 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33908.log.gz 2023-11-16T01:52:20.284 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33663.log 2023-11-16T01:52:20.284 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34940.log: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.34940.log.gz 2023-11-16T01:52:20.284 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-16T01:52:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34815.log.gz 2023-11-16T01:52:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35115.log 2023-11-16T01:52:20.285 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33663.log.gz 2023-11-16T01:52:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 /var/log/ceph/ceph.log: --verbose -- /var/log/ceph/ceph-client.admin.30833.log 2023-11-16T01:52:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35341.log 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35115.log.gz 2023-11-16T01:52:20.285 INFO:teuthology.orchestra.run.smithi136.stderr: 2023-11-16T01:52:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.30833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30833.log.gz 2023-11-16T01:52:20.285 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-16T01:52:20.286 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34451.log 2023-11-16T01:52:20.286 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35341.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-11-16T01:52:20.286 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35341.log.gz 2023-11-16T01:52:20.286 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34503.log 2023-11-16T01:52:20.286 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34451.log.gz 2023-11-16T01:52:20.286 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34323.log 2023-11-16T01:52:20.286 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33814.log 2023-11-16T01:52:20.286 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34503.log.gz 2023-11-16T01:52:20.287 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34784.log 2023-11-16T01:52:20.287 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34323.log.gz 2023-11-16T01:52:20.287 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33814.log.gz 2023-11-16T01:52:20.287 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34545.log 2023-11-16T01:52:20.287 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34784.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-11-16T01:52:20.287 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34784.log.gz 2023-11-16T01:52:20.287 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30808.log 2023-11-16T01:52:20.287 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34545.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34545.log.gz 2023-11-16T01:52:20.288 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34004.log 2023-11-16T01:52:20.288 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33689.log 2023-11-16T01:52:20.288 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.30808.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30808.log.gz 2023-11-16T01:52:20.288 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34004.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34004.log.gz 2023-11-16T01:52:20.288 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35065.log 2023-11-16T01:52:20.288 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35168.log 2023-11-16T01:52:20.288 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33689.log.gz 2023-11-16T01:52:20.288 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35246.log 2023-11-16T01:52:20.289 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35065.log.gz 2023-11-16T01:52:20.289 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35168.log.gz 2023-11-16T01:52:20.290 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35015.log 2023-11-16T01:52:20.290 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34403.log 2023-11-16T01:52:20.290 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34211.log 2023-11-16T01:52:20.290 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35015.log.gz 2023-11-16T01:52:20.290 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35246.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35246.log.gz 2023-11-16T01:52:20.292 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34403.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34403.log.gz 2023-11-16T01:52:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34211.log.gz 2023-11-16T01:52:20.293 INFO:teuthology.orchestra.run.smithi136.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-11-16T01:52:20.295 INFO:teuthology.orchestra.run.smithi136.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-16T01:52:20.307 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35936.log /var/log/ceph/ceph-client.admin.35231.log: 2023-11-16T01:52:20.307 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35231.log.gz 2023-11-16T01:52:20.307 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34235.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34235.log.gz -- replaced with /var/log/ceph/ceph-client.admin.34289.log.gz 2023-11-16T01:52:20.307 INFO:teuthology.orchestra.run.smithi083.stderr: 2023-11-16T01:52:20.307 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34399.log 2023-11-16T01:52:20.308 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34699.log 2023-11-16T01:52:20.308 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35936.log.gz 2023-11-16T01:52:20.308 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 /var/log/ceph/ceph-client.admin.34399.log: --verbose -- /var/log/ceph/ceph-client.admin.35422.log 2023-11-16T01:52:20.308 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34399.log.gz 2023-11-16T01:52:20.308 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34699.log.gz 2023-11-16T01:52:20.308 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35530.log 2023-11-16T01:52:20.308 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34907.log 2023-11-16T01:52:20.308 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35422.log.gz 2023-11-16T01:52:20.309 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35639.log 2023-11-16T01:52:20.309 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35530.log.gz 2023-11-16T01:52:20.309 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34154.log 2023-11-16T01:52:20.309 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34907.log.gz 2023-11-16T01:52:20.309 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33479.log 2023-11-16T01:52:20.309 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35639.log.gz 2023-11-16T01:52:20.309 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34073.log 2023-11-16T01:52:20.309 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34154.log.gz 2023-11-16T01:52:20.309 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34019.log 2023-11-16T01:52:20.310 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33479.log.gz 2023-11-16T01:52:20.310 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34643.log 2023-11-16T01:52:20.310 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34073.log.gz 2023-11-16T01:52:20.310 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33641.log 2023-11-16T01:52:20.310 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34019.log.gz 2023-11-16T01:52:20.310 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35666.log 2023-11-16T01:52:20.310 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34643.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34643.log.gz 2023-11-16T01:52:20.310 INFO:teuthology.orchestra.run.smithi083.stderr:gzip /var/log/ceph/ceph-client.admin.33641.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34208.log 2023-11-16T01:52:20.311 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33641.log.gz 2023-11-16T01:52:20.311 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34589.log 2023-11-16T01:52:20.311 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35666.log.gz 2023-11-16T01:52:20.311 INFO:teuthology.orchestra.run.smithi083.stderr:gzip /var/log/ceph/ceph-client.admin.34208.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35150.log 2023-11-16T01:52:20.311 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34208.log.gz 2023-11-16T01:52:20.311 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34810.log 2023-11-16T01:52:20.311 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34589.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34589.log.gz 2023-11-16T01:52:20.311 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34046.log 2023-11-16T01:52:20.312 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35150.log.gz 2023-11-16T01:52:20.312 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34100.log 2023-11-16T01:52:20.312 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34810.log.gz 2023-11-16T01:52:20.312 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34934.log 2023-11-16T01:52:20.312 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34046.log.gz 2023-11-16T01:52:20.312 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36017.log 2023-11-16T01:52:20.312 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35909.log 2023-11-16T01:52:20.312 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34100.log.gz 2023-11-16T01:52:20.313 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35123.log 2023-11-16T01:52:20.313 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36017.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34934.log.gz 2023-11-16T01:52:20.313 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36017.log.gz 2023-11-16T01:52:20.313 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36260.log 2023-11-16T01:52:20.313 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35909.log.gz 2023-11-16T01:52:20.313 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35855.log 2023-11-16T01:52:20.313 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35123.log.gz 2023-11-16T01:52:20.313 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35557.log 2023-11-16T01:52:20.313 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36260.log.gz 2023-11-16T01:52:20.314 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-11-16T01:52:20.314 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35855.log.gz 2023-11-16T01:52:20.314 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35096.log 2023-11-16T01:52:20.315 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35557.log.gz 2023-11-16T01:52:20.315 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35368.log 2023-11-16T01:52:20.315 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-16T01:52:20.315 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35096.log.gz 2023-11-16T01:52:20.315 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35990.log 2023-11-16T01:52:20.315 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35368.log.gz 2023-11-16T01:52:20.315 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36098.log 2023-11-16T01:52:20.315 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.35990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34562.log 2023-11-16T01:52:20.316 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35990.log.gz 2023-11-16T01:52:20.316 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36098.log.gz 2023-11-16T01:52:20.316 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35720.log 2023-11-16T01:52:20.316 INFO:teuthology.orchestra.run.smithi083.stderr:gzip /var/log/ceph/ceph-client.admin.34562.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33668.log 2023-11-16T01:52:20.316 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34562.log.gz 2023-11-16T01:52:20.316 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35069.log 2023-11-16T01:52:20.316 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35720.log.gz 2023-11-16T01:52:20.316 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34670.log 2023-11-16T01:52:20.316 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33668.log.gz 2023-11-16T01:52:20.317 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33399.log 2023-11-16T01:52:20.317 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35069.log.gz 2023-11-16T01:52:20.317 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35611.log 2023-11-16T01:52:20.317 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34670.log: 94.2% 0.0% -- replaced with /var/log/ceph/ceph.audit.log.gz -- replaced with /var/log/ceph/ceph-client.admin.34670.log.gz 2023-11-16T01:52:20.317 INFO:teuthology.orchestra.run.smithi083.stderr: 2023-11-16T01:52:20.317 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34782.log 2023-11-16T01:52:20.317 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33399.log.gz 2023-11-16T01:52:20.317 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36179.log 2023-11-16T01:52:20.318 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35611.log.gz 2023-11-16T01:52:20.318 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34370.log 2023-11-16T01:52:20.318 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34782.log.gz 2023-11-16T01:52:20.318 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34880.log 2023-11-16T01:52:20.318 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36179.log.gz 2023-11-16T01:52:20.318 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33722.log 2023-11-16T01:52:20.318 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34370.log.gz 2023-11-16T01:52:20.318 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36044.log 2023-11-16T01:52:20.319 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34880.log.gz 2023-11-16T01:52:20.319 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35747.log 2023-11-16T01:52:20.319 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33722.log.gz 2023-11-16T01:52:20.319 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36206.log 2023-11-16T01:52:20.319 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36044.log.gz 2023-11-16T01:52:20.319 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34316.log 2023-11-16T01:52:20.320 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35747.log: /var/log/ceph/ceph-client.admin.36206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35747.log.gz 2023-11-16T01:52:20.320 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34427.log 2023-11-16T01:52:20.320 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36206.log.gz 2023-11-16T01:52:20.320 INFO:teuthology.orchestra.run.smithi083.stderr:gzip /var/log/ceph/ceph-client.admin.34316.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35177.log 2023-11-16T01:52:20.320 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34316.log.gz 2023-11-16T01:52:20.320 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 /var/log/ceph/ceph-client.admin.34427.log: --verbose -- /var/log/ceph/ceph-client.admin.35449.log 2023-11-16T01:52:20.320 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34427.log.gz 2023-11-16T01:52:20.320 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33533.log 2023-11-16T01:52:20.321 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35177.log.gz 2023-11-16T01:52:20.321 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33560.log 2023-11-16T01:52:20.321 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35449.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35449.log.gz 2023-11-16T01:52:20.321 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 /var/log/ceph/ceph-client.admin.33533.log: --verbose -- /var/log/ceph/ceph-client.admin.35828.log 2023-11-16T01:52:20.321 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33533.log.gz 2023-11-16T01:52:20.321 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33560.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-16T01:52:20.321 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33560.log.gz 2023-11-16T01:52:20.321 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30942.log 2023-11-16T01:52:20.322 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35828.log.gz 2023-11-16T01:52:20.322 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33992.log 2023-11-16T01:52:20.322 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34262.log 2023-11-16T01:52:20.322 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30942.log.gz 2023-11-16T01:52:20.322 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35395.log 2023-11-16T01:52:20.322 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33992.log.gz 2023-11-16T01:52:20.322 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35287.log 2023-11-16T01:52:20.322 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34262.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.34262.log.gz-- 2023-11-16T01:52:20.323 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35693.log 2023-11-16T01:52:20.323 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35395.log.gz 2023-11-16T01:52:20.323 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30726.log 2023-11-16T01:52:20.323 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35287.log.gz 2023-11-16T01:52:20.323 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34454.log 2023-11-16T01:52:20.323 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35693.log: /var/log/ceph/ceph-client.admin.30726.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27084.log 2023-11-16T01:52:20.323 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30726.log.gz 2023-11-16T01:52:20.323 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35693.log.gz 2023-11-16T01:52:20.324 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36312.log 2023-11-16T01:52:20.324 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34454.log.gz 2023-11-16T01:52:20.324 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.tmp-client.admin.27084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33938.log 2023-11-16T01:52:20.324 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27084.log.gz 2023-11-16T01:52:20.324 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34753.log 2023-11-16T01:52:20.324 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36312.log.gz/var/log/ceph/ceph-client.admin.33938.log: 2023-11-16T01:52:20.325 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36287.log 2023-11-16T01:52:20.325 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33938.log.gz 2023-11-16T01:52:20.325 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33425.log 2023-11-16T01:52:20.325 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34753.log.gz 2023-11-16T01:52:20.325 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34726.log 2023-11-16T01:52:20.325 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36287.log.gz 2023-11-16T01:52:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33425.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33425.log.gz 2023-11-16T01:52:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35314.log 2023-11-16T01:52:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33749.log 2023-11-16T01:52:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34726.log.gz 2023-11-16T01:52:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34127.log 2023-11-16T01:52:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35314.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.35314.log.gz-- 2023-11-16T01:52:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35774.log 2023-11-16T01:52:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33749.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33749.log.gz 2023-11-16T01:52:20.327 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34508.log 2023-11-16T01:52:20.327 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34127.log.gz 2023-11-16T01:52:20.327 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33695.log 2023-11-16T01:52:20.327 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35774.log: /var/log/ceph/ceph-client.admin.34508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35774.log.gz 2023-11-16T01:52:20.327 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35042.log 2023-11-16T01:52:20.327 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34508.log.gz 2023-11-16T01:52:20.328 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34616.log 2023-11-16T01:52:20.328 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33695.log.gz 2023-11-16T01:52:20.328 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33614.log 2023-11-16T01:52:20.328 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35042.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36152.log 2023-11-16T01:52:20.328 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34616.log: -- replaced with /var/log/ceph/ceph-client.admin.35042.log.gz 2023-11-16T01:52:20.328 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34616.log.gz 2023-11-16T01:52:20.328 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33614.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.33614.log.gz 2023-11-16T01:52:20.328 INFO:teuthology.orchestra.run.smithi083.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.34481.log 2023-11-16T01:52:20.329 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34535.log 2023-11-16T01:52:20.329 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36071.log 0.0% 2023-11-16T01:52:20.329 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36152.log.gz 2023-11-16T01:52:20.329 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34481.log.gz 2023-11-16T01:52:20.329 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 /var/log/ceph/ceph-client.admin.34535.log: --verbose -- /var/log/ceph/ceph-client.admin.33857.log 2023-11-16T01:52:20.329 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34535.log.gz 2023-11-16T01:52:20.329 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33911.log 2023-11-16T01:52:20.329 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36071.log.gz 2023-11-16T01:52:20.329 INFO:teuthology.orchestra.run.smithi083.stderr:gzip /var/log/ceph/ceph-client.admin.33857.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33803.log 2023-11-16T01:52:20.330 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33857.log.gz 2023-11-16T01:52:20.330 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-16T01:52:20.330 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33911.log: 0.0%/var/log/ceph/ceph-client.admin.33803.log: -- replaced with /var/log/ceph/ceph-client.admin.33911.log.gz 2023-11-16T01:52:20.330 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33803.log.gz 2023-11-16T01:52:20.330 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33776.log 2023-11-16T01:52:20.330 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35341.log 2023-11-16T01:52:20.330 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35584.log 2023-11-16T01:52:20.330 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33776.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33776.log.gz 2023-11-16T01:52:20.331 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35341.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-16T01:52:20.331 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35341.log.gz 2023-11-16T01:52:20.331 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35584.log: /var/log/ceph/ceph-client.admin.34853.log 2023-11-16T01:52:20.331 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35584.log.gz 2023-11-16T01:52:20.331 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35503.log 2023-11-16T01:52:20.331 INFO:teuthology.orchestra.run.smithi083.stderr:gzip /var/log/ceph/ceph-client.admin.34853.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35882.log 2023-11-16T01:52:20.331 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34853.log.gz 92.7% 2023-11-16T01:52:20.331 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33884.log 2023-11-16T01:52:20.332 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35503.log.gz 2023-11-16T01:52:20.332 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.35882.log: -- /var/log/ceph/ceph-client.admin.34343.log 2023-11-16T01:52:20.332 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35882.log.gz 2023-11-16T01:52:20.332 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2023-11-16T01:52:20.332 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33506.log 2023-11-16T01:52:20.332 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33884.log.gz 2023-11-16T01:52:20.332 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36233.log 2023-11-16T01:52:20.332 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34343.log.gz 2023-11-16T01:52:20.333 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33830.log 2023-11-16T01:52:20.333 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33506.log.gz 2023-11-16T01:52:20.333 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36233.log.gz 2023-11-16T01:52:20.333 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34961.log 2023-11-16T01:52:20.333 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35015.log 2023-11-16T01:52:20.333 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33830.log.gz 2023-11-16T01:52:20.333 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35258.log 2023-11-16T01:52:20.333 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34961.log.gz 2023-11-16T01:52:20.333 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34988.log 2023-11-16T01:52:20.334 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35015.log.gz 2023-11-16T01:52:20.334 INFO:teuthology.orchestra.run.smithi083.stderr:gzip /var/log/ceph/ceph-client.admin.35258.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35476.log 2023-11-16T01:52:20.334 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35258.log.gz 2023-11-16T01:52:20.334 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33587.log 2023-11-16T01:52:20.334 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.34988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34988.log.gz 2023-11-16T01:52:20.334 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30850.log 2023-11-16T01:52:20.334 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35476.log.gz 2023-11-16T01:52:20.334 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.33587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33587.log.gz 2023-11-16T01:52:20.335 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-16T01:52:20.335 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.30850.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30850.log.gz 2023-11-16T01:52:20.335 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.2.log: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-11-16T01:52:20.470 INFO:teuthology.orchestra.run.smithi083.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-16T01:52:20.742 INFO:teuthology.orchestra.run.smithi136.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-11-16T01:52:20.763 INFO:teuthology.orchestra.run.smithi083.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-16T01:52:20.765 INFO:teuthology.orchestra.run.smithi136.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-11-16T01:52:20.778 INFO:teuthology.orchestra.run.smithi136.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-11-16T01:52:20.780 INFO:teuthology.orchestra.run.smithi136.stderr: 2023-11-16T01:52:20.780 INFO:teuthology.orchestra.run.smithi136.stderr:real 0m0.600s 2023-11-16T01:52:20.780 INFO:teuthology.orchestra.run.smithi136.stderr:user 0m1.732s 2023-11-16T01:52:20.780 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m0.110s 2023-11-16T01:52:20.824 INFO:teuthology.orchestra.run.smithi083.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-16T01:52:20.834 INFO:teuthology.orchestra.run.smithi083.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-16T01:52:20.834 INFO:teuthology.orchestra.run.smithi083.stderr: 2023-11-16T01:52:20.835 INFO:teuthology.orchestra.run.smithi083.stderr:real 0m0.657s 2023-11-16T01:52:20.835 INFO:teuthology.orchestra.run.smithi083.stderr:user 0m2.126s 2023-11-16T01:52:20.835 INFO:teuthology.orchestra.run.smithi083.stderr:sys 0m0.096s 2023-11-16T01:52:20.836 INFO:tasks.ceph:Archiving logs... 2023-11-16T01:52:20.836 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694/remote/smithi083/log 2023-11-16T01:52:20.837 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-16T01:52:21.347 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694/remote/smithi136/log 2023-11-16T01:52:21.348 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-16T01:52:21.714 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-16T01:52:21.729 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-16T01:52:21.729 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-16T01:52:21.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-16T01:52:21.802 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-16T01:52:21.832 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-16T01:52:21.833 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T01:52:21.837 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T01:52:21.859 INFO:teuthology.orchestra.run.smithi083.stdout: remote refid st t when poll reach delay offset jitter 2023-11-16T01:52:21.859 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================== 2023-11-16T01:52:21.859 INFO:teuthology.orchestra.run.smithi083.stdout:*hv01.front.sepi 44.4.53.2 2 u 3 64 177 0.059 -3.252 0.217 2023-11-16T01:52:21.860 INFO:teuthology.orchestra.run.smithi083.stdout:+hv02.front.sepi 63.231.80.2 3 u 10 64 177 0.085 -0.813 0.499 2023-11-16T01:52:21.860 INFO:teuthology.orchestra.run.smithi083.stdout:+hv03.front.sepi 74.6.168.72 3 u 6 64 177 0.091 0.461 0.281 2023-11-16T01:52:21.860 INFO:teuthology.orchestra.run.smithi083.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-16T01:52:21.861 INFO:teuthology.orchestra.run.smithi136.stdout: remote refid st t when poll reach delay offset jitter 2023-11-16T01:52:21.861 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================== 2023-11-16T01:52:21.861 INFO:teuthology.orchestra.run.smithi136.stdout:+hv01.front.sepi 44.4.53.2 2 u 7 64 177 0.082 -3.456 1.313 2023-11-16T01:52:21.862 INFO:teuthology.orchestra.run.smithi136.stdout:*hv02.front.sepi 63.231.80.2 3 u 10 64 177 0.074 1.321 1.533 2023-11-16T01:52:21.862 INFO:teuthology.orchestra.run.smithi136.stdout:+hv03.front.sepi 74.6.168.72 3 u 6 64 177 0.078 1.956 1.513 2023-11-16T01:52:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-16T01:52:21.862 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-16T01:52:21.872 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-16T01:52:21.873 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-16T01:52:21.917 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-16T01:52:21.966 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-16T01:52:22.015 INFO:teuthology.task.internal:Duration was 771.439394 seconds 2023-11-16T01:52:22.016 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-16T01:52:22.065 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-16T01:52:22.065 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-16T01:52:22.068 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-16T01:52:22.123 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-16T01:52:22.123 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com 2023-11-16T01:52:22.124 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-16T01:52:22.137 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2023-11-16T01:52:22.137 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-16T01:52:22.153 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-16T01:52:22.153 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T01:52:22.183 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T01:52:22.218 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-16T01:52:22.219 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-16T01:52:22.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-16T01:52:22.322 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-16T01:52:22.334 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-16T01:52:22.334 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-16T01:52:22.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-16T01:52:22.369 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-16T01:52:22.380 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-16T01:52:22.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-16T01:52:22.408 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core 2023-11-16T01:52:22.410 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2023-11-16T01:52:22.438 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-16T01:52:22.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:52:22.444 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-16T01:52:22.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:52:22.486 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-16T01:52:22.496 INFO:teuthology.task.internal:Transferring archived files... 2023-11-16T01:52:22.497 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694/remote/smithi083 2023-11-16T01:52:22.497 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-16T01:52:22.519 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694/remote/smithi136 2023-11-16T01:52:22.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-16T01:52:22.543 INFO:teuthology.task.internal:Removing archive directory... 2023-11-16T01:52:22.543 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-16T01:52:22.559 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-16T01:52:22.568 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-16T01:52:22.585 INFO:teuthology.task.internal:Not uploading archives. 2023-11-16T01:52:22.585 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-16T01:52:22.611 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-16T01:52:22.611 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-16T01:52:22.615 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-16T01:52:22.619 INFO:teuthology.orchestra.run.smithi083.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 16 01:52 /home/ubuntu/cephtest 2023-11-16T01:52:22.623 INFO:teuthology.orchestra.run.smithi136.stdout: 924112 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 16 01:52 /home/ubuntu/cephtest 2023-11-16T01:52:22.623 INFO:teuthology.orchestra.run.smithi136.stdout: 1064733 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 16 01:48 /home/ubuntu/cephtest/mnt.0 2023-11-16T01:52:22.623 INFO:teuthology.orchestra.run.smithi136.stdout: 1064734 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 16 01:50 /home/ubuntu/cephtest/mnt.0/client.0 2023-11-16T01:52:22.624 INFO:teuthology.orchestra.run.smithi136.stdout: 1194447 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Nov 16 01:50 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T01:52:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:52:22.626 INFO:teuthology.orchestra.run.smithi136.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-16T01:52:22.626 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi136 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-11-16T01:52:22.634 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-16T01:52:22.659 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-16T01:52:22.715 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi136 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-11-16T01:52:22.715 INFO:teuthology.nuke:Checking targets against current locks 2023-11-16T01:52:22.749 DEBUG:teuthology.nuke:shortname: smithi083 2023-11-16T01:52:22.749 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T01:52:22.773 DEBUG:teuthology.nuke:shortname: smithi136 2023-11-16T01:52:22.774 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T01:52:22.778 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 01:30:26.791767', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-11-16T01:52:22.808 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459694', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 01:30:26.793190', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-11-16T01:52:22.841 INFO:teuthology.orchestra.console.smithi083:Power off 2023-11-16T01:52:22.842 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-16T01:52:22.866 DEBUG:teuthology.orchestra.console.smithi083:power off output: Chassis Power Control: Down/Off 2023-11-16T01:52:22.866 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T01:52:22.886 INFO:teuthology.orchestra.console.smithi136:Power off 2023-11-16T01:52:22.886 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-16T01:52:22.899 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2023-11-16T01:52:22.911 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off 2023-11-16T01:52:22.911 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T01:52:22.940 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2023-11-16T01:52:26.899 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T01:52:26.941 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T01:52:27.010 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2023-11-16T01:52:27.055 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2023-11-16T01:52:31.012 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T01:52:31.056 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T01:52:31.123 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2023-11-16T01:52:31.169 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2023-11-16T01:52:31.269 INFO:teuthology.orchestra.console.smithi136:Power off completed 2023-11-16T01:52:35.124 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T01:52:35.238 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off 2023-11-16T01:52:35.339 INFO:teuthology.orchestra.console.smithi083:Power off completed 2023-11-16T01:52:35.444 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_workunits} duration: 771.4393935203552 failure_reason: 'Command failed (workunit test mon/crush_ops.sh) on smithi136 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ed3c8e926b9de93026c8a7c1d86be5d6821b52ac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=89688c83dd364ce2aa0daaf0410e9434 status: fail success: false 2023-11-16T01:52:35.445 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-16T01:52:35.569 INFO:teuthology.run:FAIL